各位IT大虾，求助

beysup

要读入400多MB的文本文件，因为涉及到encoding的问题，要通过StreamReader来读入（.NET），在开辟buffer时候，抛出out of memory异常，因为物理内存不够。如何在开辟buffer的时候使用虚拟内存，是否可以通过调用API来实现？谢谢

hoopoos

我不熟.Net，粗略建议

1。不要用buffer一次全部装载整个文件，用low level IO API，File inputstream之类的，一个字节一个字节的读

2。增加进程的内存参数（虚拟内存是OS管理的，不需要application指定）

3。调用IO前显式请求垃圾回收，释放内存

beysup

谢谢ls

我要处理的数据是Big5 encoding，如果用byte stream，那么这个encoding的信息无法保存，只能用16bits的char来保存，在.NET下，char是16bits的unicode，所以只能选择StreamReader，因为只有这个才可以指定encoding。

问题关键还是buffer，如果不一次装载整个文件，那么这个buffer开多大比较合适？因为这个文件是corpus collection，无法确定每个corpus的长度，而且分批次读，还涉及索引以及效率问题。

Coolioo

My 2 cents

1. The data file does not contain encoding info, so there is no such a thing called "Loosing the encoding info".

2. Virtual memory is managed by OS, you are not supposed to touch it.

3. Set the buffer lower, bufferred stream will load the data from the file system to memory buffer automatically. 10M buffer is enough for a 400M file.

4. Paste you code here so that we can know what you are doing exactly....

[ 本帖最后由 Coolioo 于 9-11-2007 18:30 编辑 ]

Coolioo

one Big5 character is fixed 16 bits, you can read two bytes from a stream and convert it to a big5 character...

But the better practice is to use StreamReader with Big5 encoding....

test001

There is one thing in computer is '\0', this means end of a line.
Just read in one line each time till EOF(end of file) and decode it.

Coolioo

\0 only exits in C/C++, C# and Java has no such a concept

Coolioo

C# Code

// --- Write a 400M file containing one big5 character 陸 ---
StreamWriter bigWriter = new StreamWriter(@"big5.txt", false, Encoding.GetEncoding("big5"));
for (int i = 0; i < 209715200; i++) bigWriter.Write("陸");
bigWriter.Close();
Console.WriteLine("Writing Done");

// --- Read big5 characters from the 400M file  ---
StreamReader bigReader = new StreamReader(@"big5.txt", Encoding.GetEncoding("big5"));
// - print out the first big character -
char[] chars = new char[1];
bigReader.Read(chars, 0, 1);
Console.WriteLine(chars);
// - Read the rest and print the total number of characters read -
int counter = 1;
while (-1 != bigReader.Read()) counter++;
bigReader.Close();
Console.WriteLine("Reading Done: " + counter);

[ 本帖最后由 Coolioo 于 9-11-2007 20:18 编辑 ]

jl162401

牛啊！！！

beysup

问题已经解决了，感谢ls几位大虾。