Let’s take a look at the simplest aggregate calculation, such as calculating the sum of amount in orders.txt. Different types of calculation need different calculation process. Using high-level language programming to calculate, how to write the calculation process is related to the specific calculation type. High-level languages(Take Java as an example) The first line in the file is the column name, with a total of 10 million lines of data. The file to be used in this paper orders.txt has five columns: orderkey, orderdate, state, quantity and amount. For other types of file, except the way of reading data is different, the processing idea after reading is similar to text file. Taking text file as an example, this paper introduces the characteristics of the above methods for large file calculation in turn. The file data is imported into the database and processed by SQL 3. Conventional high-level programming languages, such as Java, C/C++, C#, Basic, etc. The program languages that can be used to process large files are as follows: 1. Among them, text (txt or CSV) is the most common. There are many types of large files, such as text files, Excel files, XML files, JSON files, HTTP files. ![]() Finally, the batch processing results need to be properly summarized according to different calculation types, which is much more complicated than the processing of small file. Even if the program is written, a large file must be read in batches for calculation and processing. In this case, direct use of desktop data tools (such as Excel) is powerless, often need to write a program to deal with it. What is a large file? A large file is a file that is too large to be read in at one time because of insufficient computer memory.
0 Comments
Leave a Reply. |