Hi,
This is vinod.
i have tried to read a 10 lacks records with column count 14 from the database at a single time. But my application shows OUT OF MEMORY error.
Any help?
my id : <id removed per posting guidelines>
Thanks,
Vinod
15 8708
Hi,
This is vinod.
i have tried to read a 10 lacks records with column count 14 from the database at a single time. But my application shows OUT OF MEMORY error.
Any help?
my id : <id removed as per posting guidelines>
Thanks,
Vinod
How much is 'a 10 lacks'?
kind regards,
Jos
10,00,000
Thanks,
vinod
So that's one million records with 14 columns each? It depends a bit on the length
of the columns but I can see that you get an OutOfMemoryError ... there are a
few -X flags that increase the JVM's heap size but I think that's just postponing
the execution. What do you want to load that many records in memory at once?
kind regards,
Jos
So that's one million records with 14 columns each? It depends a bit on the length
of the columns but I can see that you get an OutOfMemoryError ... there are a
few -X flags that increase the JVM's heap size but I think that's just postponing
the execution. What do you want to load that many records in memory at once?
kind regards,
Jos
Thanks for ur response Jos.
I need to import data from one database to another database. Please give me some way to avoid this Outof Memory error.
Regards,
Vinod
Thanks for ur response Jos.
I need to import data from one database to another database. Please give me some way to avoid this Outof Memory error.
Regards,
Vinod
You might try to specify the Statement.setFetchSize() on your Statement
kind regards,
Jos
You might try to specify the Statement.setFetchSize() on your Statement
kind regards,
Jos
I have tried this setFetchSize() method. But I got error message(Invalid fetch size).
Thanks,
Vinod
I have tried this setFetchSize() method. But I got error message(Invalid fetch size).
Thanks,
Vinod
So what happens if you experiment a bit with different fetch sizes?
kind regards,
Jos
So what happens if you experiment a bit with different fetch sizes?
kind regards,
Jos
Hi,
I got the following error.
rs.setFetchSize(50);
java.sql.SQLException: Invalid Fetch Size
Thanks,
Vinod
Hi,
I got the following error.
rs.setFetchSize(50);
java.sql.SQLException: Invalid Fetch Size
Thanks,
Vinod
Can you show us a bit of relevant code please? What database are you using?
kind regards,
Jos
Hi
The ResultSet is having a limit of 1000 results. You can not access unlimited number of records from a singlr sql query. So you are getting errors
The ResultSet is having a limit of 1000 results.
That all depends on the database vendor and its JDBC driver.
kind regards,
Jos
That all depends on the database vendor and its JDBC driver.
kind regards,
Jos
Hi,
I have tried to read 3,00,000 records. I was able to do this.
Please advise. Is any other way to perform this.
Thanks,
Vinod
Hi,
I have tried to read 3,00,000 records. I was able to do this.
Please advise. Is any other way to perform this.
Thanks,
Vinod
I don't know if you followed an other identical thread but my observations are:
1) users are not willing to scroll through so much data. They want it in reasonable
chunks through which they can navigate.
2) you can't collect that much database because of memory constraints.
So my tip is to 'page' through all the data. Depending on your database and its
SQL dialect you might check the 'offset/limit' statement parts or the 'numrows'
statement part or whatever your database supports.
A few buttons for navigational purposes and just a bit of data in the Session
object can do the rest. Don't try to retrieve and show all the data in one piece.
kind regards,
Jos
I don't know if you followed an other identical thread but my observations are:
1) users are not willing to scroll through so much data. They want it in reasonable
chunks through which they can navigate.
2) you can't collect that much database because of memory constraints.
So my tip is to 'page' through all the data. Depending on your database and its
SQL dialect you might check the 'offset/limit' statement parts or the 'numrows'
statement part or whatever your database supports.
A few buttons for navigational purposes and just a bit of data in the Session
object can do the rest. Don't try to retrieve and show all the data in one piece.
kind regards,
Jos
thanks Jos. Let me try
vinod
Please use this this will help you in storing the bulk data without memory dependency
Statement stmt = con.createStatement(java.sql.ResultSet.TYPE_FORWAR D_ONLY,java.sql.ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(Integer.MIN_VALUE);
Sign in to post your reply or Sign up for a free account.
Similar topics
by: Steve |
last post by:
Hi,
I have a web application where there are a number of columns a user can
sort on (price, item etc). The database itself is HUGE (around 6 million
rows) and even with all the indexing it slows...
|
by: Rainer Collet |
last post by:
hi!
i tested several php database abstraction layers (db, mdb(2), creole,
adodb, etc), but i always missed one really important feature:
i need a method for a limited select which gives me the...
|
by: RoyalScorpion |
last post by:
hi guys, i get an updatable resultset from a connection then used it
to insert a new row but the result set doesn't chane after insertion,
i mean the no of rows before insertion is the same after...
|
by: Karam Chand |
last post by:
Greetings
I have a table with the following table structure -
Field Type Collation Null Key
Default Extra
------- ------------- ----------------- ------
------ ...
|
by: Karam Chand |
last post by:
Greetings
I have a table with the following table structure -
Field Type Collation Null Key
Default Extra
------- ------------- ----------------- ------
------ ...
|
by: _link98 |
last post by:
Problem: java ResultSet cursor from SQL/PL stored-procedure is
FORWARD_ONLY.
Is it possible to have ResultSet cursors from SQL/PL procedures to
scroll forward and backwards? Perhaps I am missing...
|
by: natG |
last post by:
Hi;
In db2 8.2, how do I limit a query resultset to x rows?
Thanks,
nat
|
by: Garth Wells |
last post by:
I use a filtered SELECT to populate the SQLDataReader (rdr) with
a filename and a blob (pdf). I then use File.WriteAllBytes to write
each pdf to disk.
----------------------------------------...
|
by: Sl1ver |
last post by:
That is my code, if i put a updateQuery statement in there, then it updates my database, but when i query the database then it doesn't populate the resultset
public ResultSet User(String user)
...
|
by: DolphinDB |
last post by:
Tired of spending countless mintues downsampling your data? Look no further!
In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: isladogs |
last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM).
In this month's session, we are pleased to welcome back...
|
by: Vimpel783 |
last post by:
Hello!
Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
|
by: ArrayDB |
last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
|
by: PapaRatzi |
last post by:
Hello,
I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
|
by: CloudSolutions |
last post by:
Introduction:
For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
|
by: Shællîpôpï 09 |
last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome former...
| |