PetrGlad/python-prevayler

Define pickle protocol used as constant and default to ASCII

mbucc opened this issue · 0 comments

I saw no speed difference in logging 40,000 transactions using the ASCII pickle protocol.

With an ASCII log file, you have some hope if you need to manually dig through the transaction history. It's also grep'able. In general, text log files are a good thing.

If file size is a concern, we should Gzip log files. (Probably should do that anyway---they get big quick.)

diff -r 687f471135fa topics/tornado/pv/core.py
--- a/topics/tornado/pv/core.py Wed Sep 28 08:02:16 2011 -0400
+++ b/topics/tornado/pv/core.py Wed Sep 28 08:08:27 2011 -0400
@@ -10,6 +10,8 @@
 import re
 from pv.util import baseN, NUMERALS

+#PICKLE_PROTOCOL = pickle.HIGHEST_PROTOCOL
+PICKLE_PROTOCOL = 0    # ASCII

 class Log(object):
     """
@@ -83,12 +85,12 @@

     def put(self, value):
         self.serialId += 1
-        pickle.dump(value, self.logFile, pickle.HIGHEST_PROTOCOL)
+        pickle.dump(value, self.logFile, PICKLE_PROTOCOL)

     def putSnapshot(self, root):
         # TODO refine error handling 
         snapshotFile = open(self.makeSnapshotName(self.serialId), 'ab')        
-        pickle.dump(root, snapshotFile, pickle.HIGHEST_PROTOCOL)
+        pickle.dump(root, snapshotFile, PICKLE_PROTOCOL)
         snapshotFile.close()        
         self.logRotate(self.serialId)