.load_file()

Today I split .historical into three parts. .historical, ._get_data(), and ._load_from_csv(). It wasn't particularly difficult, but it was in effort to make a .load_file() in the Stock class that would load high frequency data. I spent about an two hours hunting down and trying to fix a bug that I thought was because of limitations to pythons f.read() function. Turns out, I was just passing the wrong parameter between functions. I thought that was the first thing I looked for. Maybe I did, but was just wrong, in any case, it works now.

EWrapper and EClientSocket

I am very close to getting my SageEWrapper (my implementation of EWrapper) to provide some necessary historical data. I figured out how the sample application gets it's historical data and found this table about the types of requests you can do. After I finish my wrapper class, I'm going to add the apache xml-rpc library to the project and start getting that up and running.


TIME PERIOD BAR SIZE
1 Y 1 day
6 M 1 day
3 M 1 day
1 M 1 day, 1 hour
1 W 1 day, 1 hour, 30 mins, 15 mins
2 D 1 hour, 30 mins, 15 mins, 3 mins, 2 mins, 1 min
1 D 1 hour, 30 mins, 15 mins, 5 mins 3 mins, 2 mins, 1 min, 30 secs
14400 S (4 hrs) 1 hour, 30 mins, 15 mins, 5 mins 3 mins, 2 mins, 1 min, 30 secs, 15 secs
7200 S (2 hrs) 1 hour, 30 mins, 15 mins, 5mins 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs
3600 S (1 hr) 15 mins, 5 mins 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs
1800 S (30 mins) 15 mins, 5 mins 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs
960 S (15 mins.) 5 mins 3 mins, 2 mins, 1 min, 30 secs, 15 secs 5 secs 1 secs
300 S (5 mins) 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs
60 S ( 1 min) 30 secs, 15 secs, 5 secs, 1 secs

S (for seconds), D (for days), M (for months), W (for weeks), and Y.

Just from the sample client from IB, here's some high-frequency stock data (30 - second bars for a period of 1 hour, ending at 7:58 AM GMT)


id=1 date = 20080717 23:58:52 open=534.49 high=534.49 low=534.49 close=534.49 volume=1 count=1 WAP=534.49 hasGaps=false
id=1 date = 20080717 23:59:00 open=534.56 high=534.56 low=534.56 close=534.56 volume=1 count=1 WAP=534.56 hasGaps=false
id=1 date = 20080717 23:59:30 open=533.72 high=533.85 low=533.71 close=533.71 volume=5 count=3 WAP=533.74 hasGaps=false
id=1 date = 20080718 00:00:00 open=533.71 high=534.79 low=533.71 close=534.51 volume=11 count=5 WAP=534.53 hasGaps=false
id=1 date = 20080718 00:00:30 open=534.78 high=535.12 low=534.78 close=535.12 volume=4 count=3 WAP=534.99 hasGaps=false
id=1 date = 20080718 00:01:00 open=535.21 high=535.86 low=535.17 close=535.76 volume=11 count=6 WAP=535.56 hasGaps=false
id=1 date = 20080718 00:01:30 open=536.06 high=536.06 low=536.06 close=536.06 volume=3 count=1 WAP=536.06 hasGaps=false
id=1 date = 20080718 00:02:00 open=536.2 high=536.2 low=536.17 close=536.17 volume=2 count=2 WAP=536.18 hasGaps=false
id=1 date = 20080718 00:02:30 open=535.65 high=536.4 low=535.65 close=536.04 volume=7 count=7 WAP=535.96 hasGaps=false
id=1 date = 20080718 00:03:00 open=536.13 high=536.27 low=535.57 close=536.27 volume=10 count=7 WAP=536.06 hasGaps=false
id=1 date = 20080718 00:03:30 open=535.6 high=535.6 low=534.97 close=535.35 volume=9 count=4 WAP=535.28 hasGaps=false
id=1 date = 20080718 00:04:00 open=534.39 high=534.39 low=534.39 close=534.39 volume=1 count=1 WAP=534.39 hasGaps=false
id=1 date = 20080718 00:04:30 open=533.92 high=534.36 low=533.84 close=534.36 volume=10 count=7 WAP=534.06 hasGaps=false
id=1 date = 20080718 00:05:00 open=533.93 high=534.55 low=533.93 close=534.32 volume=8 count=4 WAP=534.36 hasGaps=false
id=1 date = 20080718 00:05:30 open=534.24 high=534.41 low=533.64 close=533.64 volume=6 count=6 WAP=534.07 hasGaps=false
id=1 date = 20080718 00:06:00 open=534.02 high=534.33 low=533.2 close=533.98 volume=10 count=8 WAP=533.91 hasGaps=false
id=1 date = 20080718 00:06:30 open=533.61 high=533.78 low=533.61 close=533.78 volume=8 count=5 WAP=533.67 hasGaps=false
id=1 date = 20080718 00:07:00 open=533.77 high=533.77 low=533.32 close=533.32 volume=7 count=4 WAP=533.65 hasGaps=false
id=1 date = 20080718 00:07:30 open=533.79 high=533.79 low=532.79 close=533.3 volume=3 count=3 WAP=533.29 hasGaps=false
id=1 date = 20080718 00:08:00 open=533.12 high=534.97 low=532.8 close=532.8 volume=6 count=6 WAP=533.39 hasGaps=false
id=1 date = 20080718 00:08:30 open=533.52 high=533.52 low=533.52 close=533.52 volume=2 count=1 WAP=533.52 hasGaps=false
id=1 date = 20080718 00:09:00 open=533.3 high=533.3 low=533.3 close=533.3 volume=1 count=1 WAP=533.3 hasGaps=false
id=1 date = 20080718 00:09:30 open=532.75 high=533.2 low=532.75 close=533.12 volume=8 count=5 WAP=532.9 hasGaps=false
id=1 date = 20080718 00:10:00 open=533.65 high=533.65 low=532.43 close=532.43 volume=12 count=8 WAP=532.7 hasGaps=false
id=1 date = 20080718 00:10:30 open=531.78 high=532.59 low=531.78 close=532.5 volume=16 count=7 WAP=532.31 hasGaps=false
id=1 date = 20080718 00:11:00 open=532.65 high=532.72 low=532.53 close=532.72 volume=3 count=3 WAP=532.63 hasGaps=false
id=1 date = 20080718 00:11:30 open=533.27 high=533.27 low=532.55 close=532.71 volume=12 count=8 WAP=532.89 hasGaps=false
id=1 date = 20080718 00:12:00 open=532.65 high=532.65 low=532.22 close=532.22 volume=2 count=2 WAP=532.43 hasGaps=false
id=1 date = 20080718 00:12:30 open=531.97 high=531.97 low=531.97 close=531.97 volume=1 count=1 WAP=531.97 hasGaps=false
id=1 date = 20080718 00:13:00 open=532.86 high=532.86 low=532.22 close=532.22 volume=7 count=4 WAP=532.42 hasGaps=false
id=1 date = 20080718 00:13:30 open=532.46 high=532.46 low=531.89 close=531.89 volume=5 count=3 WAP=532.32 hasGaps=false
id=1 date = 20080718 00:14:00 open=532.13 high=532.2 low=531.85 close=532.2 volume=6 count=4 WAP=532.09 hasGaps=false
id=1 date = 20080718 00:14:30 open=532.32 high=532.86 low=532.32 close=532.86 volume=4 count=3 WAP=532.62 hasGaps=false
id=1 date = 20080718 00:15:00 open=533.45 high=533.45 low=532.67 close=532.86 volume=9 count=6 WAP=532.95 hasGaps=false
id=1 date = 20080718 00:15:30 open=532.79 high=532.79 low=532.79 close=532.79 volume=1 count=1 WAP=532.79 hasGaps=false
id=1 date = 20080718 00:16:00 open=532.29 high=532.29 low=532.29 close=532.29 volume=1 count=1 WAP=532.29 hasGaps=false
id=1 date = 20080718 00:16:30 open=531.87 high=531.99 low=531.64 close=531.99 volume=5 count=4 WAP=531.8 hasGaps=false
id=1 date = 20080718 00:17:00 open=532.07 high=532.27 low=532.01 close=532.01 volume=4 count=4 WAP=532.13 hasGaps=false
id=1 date = 20080718 00:17:30 open=532.3 high=532.3 low=531.78 close=531.94 volume=11 count=8 WAP=532.07 hasGaps=false
id=1 date = 20080718 00:18:00 open=531.81 high=531.81 low=531.58 close=531.6 volume=9 count=4 WAP=531.65 hasGaps=false
id=1 date = 20080718 00:18:30 open=531.62 high=531.62 low=531.43 close=531.43 volume=2 count=2 WAP=531.52 hasGaps=false
id=1 date = 20080718 00:19:00 open=531.71 high=531.71 low=531.05 close=531.05 volume=9 count=6 WAP=531.31 hasGaps=false
id=1 date = 20080718 00:19:30 open=531.36 high=531.78 low=531.11 close=531.57 volume=8 count=8 WAP=531.52 hasGaps=false
id=1 date = 20080718 00:20:00 open=531.85 high=532.27 low=531.5 close=531.5 volume=14 count=7 WAP=531.91 hasGaps=false
id=1 date = 20080718 00:20:30 open=533.0 high=533.0 low=532.13 close=532.41 volume=10 count=5 WAP=532.51 hasGaps=false
id=1 date = 20080718 00:21:00 open=531.79 high=532.49 low=531.79 close=532.49 volume=11 count=4 WAP=532.23 hasGaps=false
id=1 date = 20080718 00:21:30 open=532.56 high=532.56 low=532.19 close=532.19 volume=2 count=2 WAP=532.37 hasGaps=false
id=1 date = 20080718 00:22:00 open=532.38 high=532.73 low=531.89 close=531.89 volume=7 count=5 WAP=532.4 hasGaps=false
id=1 date = 20080718 00:22:30 open=532.06 high=532.06 low=531.5 close=532.05 volume=5 count=3 WAP=531.72 hasGaps=false
id=1 date = 20080718 00:23:00 open=531.9 high=531.9 low=531.51 close=531.51 volume=6 count=5 WAP=531.75 hasGaps=false
id=1 date = 20080718 00:23:30 open=532.5 high=532.5 low=532.5 close=532.5 volume=1 count=1 WAP=532.5 hasGaps=false
id=1 date = 20080718 00:24:00 open=532.17 high=532.58 low=532.17 close=532.43 volume=12 count=7 WAP=532.38 hasGaps=false
id=1 date = 20080718 00:24:30 open=532.37 high=532.37 low=532.15 close=532.15 volume=2 count=2 WAP=532.26 hasGaps=false
id=1 date = 20080718 00:25:00 open=532.88 high=533.25 low=531.99 close=531.99 volume=9 count=5 WAP=532.85 hasGaps=false
id=1 date = 20080718 00:25:30 open=532.36 high=533.36 low=532.36 close=532.89 volume=21 count=11 WAP=532.89 hasGaps=false
id=1 date = 20080718 00:26:00 open=533.15 high=533.95 low=533.15 close=533.46 volume=3 count=3 WAP=533.52 hasGaps=false
id=1 date = 20080718 00:26:30 open=533.47 high=533.47 low=532.81 close=532.81 volume=5 count=5 WAP=533.2 hasGaps=false
id=1 date = 20080718 00:27:00 open=532.56 high=533.12 low=532.56 close=532.6 volume=11 count=11 WAP=532.83 hasGaps=false
id=1 date = 20080718 00:27:30 open=532.65 high=533.1 low=532.65 close=533.04 volume=4 count=4 WAP=532.95 hasGaps=false
id=1 date = 20080718 00:28:00 open=533.48 high=533.48 low=533.19 close=533.19 volume=2 count=2 WAP=533.34 hasGaps=false
id=1 date = 20080718 00:28:30 open=533.49 high=533.49 low=533.1 close=533.1 volume=2 count=2 WAP=533.29 hasGaps=false
id=1 date = 20080718 00:29:00 open=532.98 high=533.33 low=532.94 close=532.95 volume=10 count=5 WAP=533.03 hasGaps=false
id=1 date = 20080718 00:29:30 open=533.46 high=533.46 low=533.46 close=533.46 volume=1 count=1 WAP=533.46 hasGaps=false
id=1 date = 20080718 00:30:00 open=532.63 high=533.38 low=532.51 close=532.51 volume=7 count=4 WAP=533.05 hasGaps=false
id=1 date = 20080718 00:30:30 open=531.68 high=531.68 low=531.68 close=531.68 volume=1 count=1 WAP=531.68 hasGaps=false
id=1 date = 20080718 00:31:00 open=532.2 high=532.8 low=531.65 close=532.22 volume=14 count=9 WAP=532.15 hasGaps=false
id=1 date = 20080718 00:31:30 open=532.13 high=532.19 low=531.94 close=531.94 volume=5 count=3 WAP=532.1 hasGaps=false
id=1 date = 20080718 00:32:00 open=532.39 high=532.39 low=532.39 close=532.39 volume=1 count=1 WAP=532.39 hasGaps=false
id=1 date = 20080718 00:32:30 open=532.38 high=532.38 low=532.03 close=532.03 volume=3 count=3 WAP=532.2 hasGaps=false
id=1 date = 20080718 00:33:00 open=532.36 high=532.75 low=531.85 close=532.25 volume=4 count=4 WAP=532.3 hasGaps=false
id=1 date = 20080718 00:33:30 open=532.72 high=533.02 low=532.11 close=532.79 volume=26 count=15 WAP=532.63 hasGaps=false
id=1 date = 20080718 00:34:00 open=532.76 high=532.76 low=532.76 close=532.76 volume=3 count=1 WAP=532.76 hasGaps=false
id=1 date = 20080718 00:34:30 open=532.23 high=532.48 low=531.94 close=532.48 volume=9 count=7 WAP=532.19 hasGaps=false
id=1 date = 20080718 00:35:00 open=532.92 high=532.92 low=532.81 close=532.81 volume=3 count=2 WAP=532.88 hasGaps=false
id=1 date = 20080718 00:35:30 open=533.63 high=533.63 low=533.13 close=533.13 volume=6 count=3 WAP=533.44 hasGaps=false
id=1 date = 20080718 00:36:00 open=533.34 high=533.56 low=532.92 close=533.42 volume=17 count=9 WAP=533.19 hasGaps=false
id=1 date = 20080718 00:36:30 open=533.46 high=533.46 low=532.54 close=533.34 volume=8 count=5 WAP=532.84 hasGaps=false
id=1 date = 20080718 00:37:00 open=534.32 high=534.71 low=533.96 close=534.63 volume=11 count=7 WAP=534.41 hasGaps=false
id=1 date = 20080718 00:37:30 open=534.76 high=535.05 low=534.76 close=534.95 volume=4 count=3 WAP=534.93 hasGaps=false
id=1 date = 20080718 00:38:00 open=534.9 high=535.05 low=534.62 close=534.86 volume=16 count=8 WAP=534.86 hasGaps=false
id=1 date = 20080718 00:38:30 open=534.45 high=534.98 low=534.45 close=534.98 volume=2 count=2 WAP=534.72 hasGaps=false
id=1 date = 20080718 00:39:00 open=535.19 high=535.19 low=535.06 close=535.09 volume=6 count=6 WAP=535.12 hasGaps=false
id=1 date = 20080718 00:39:30 open=535.23 high=535.36 low=534.89 close=535.17 volume=7 count=5 WAP=535.22 hasGaps=false
id=1 date = 20080718 00:40:00 open=534.85 high=535.31 low=534.85 close=535.31 volume=3 count=3 WAP=535.02 hasGaps=false
id=1 date = 20080718 00:40:30 open=535.03 high=535.2 low=534.6 close=534.94 volume=19 count=13 WAP=534.9 hasGaps=false
id=1 date = 20080718 00:41:00 open=534.66 high=534.66 low=534.66 close=534.66 volume=1 count=1 WAP=534.66 hasGaps=false
id=1 date = 20080718 00:41:30 open=534.42 high=534.71 low=534.42 close=534.71 volume=4 count=2 WAP=534.64 hasGaps=false
id=1 date = 20080718 00:42:00 open=534.48 high=534.95 low=534.31 close=534.95 volume=5 count=5 WAP=534.56 hasGaps=false
id=1 date = 20080718 00:42:30 open=535.12 high=535.38 low=534.29 close=534.29 volume=20 count=13 WAP=534.85 hasGaps=false
id=1 date = 20080718 00:43:00 open=535.26 high=535.26 low=535.13 close=535.13 volume=2 count=2 WAP=535.2 hasGaps=false
id=1 date = 20080718 00:43:30 open=535.67 high=535.68 low=535.21 close=535.21 volume=8 count=3 WAP=535.5 hasGaps=false
id=1 date = 20080718 00:44:00 open=535.27 high=535.57 low=535.27 close=535.57 volume=5 count=2 WAP=535.51 hasGaps=false
id=1 date = 20080718 00:44:30 open=535.79 high=536.22 low=535.79 close=536.22 volume=2 count=2 WAP=536.0 hasGaps=false
id=1 date = 20080718 00:45:00 open=536.08 high=536.08 low=536.08 close=536.08 volume=1 count=1 WAP=536.08 hasGaps=false
id=1 date = 20080718 00:45:30 open=535.38 high=535.85 low=535.38 close=535.53 volume=5 count=4 WAP=535.62 hasGaps=false
id=1 date = 20080718 00:46:00 open=535.78 high=536.2 low=535.78 close=535.9 volume=6 count=3 WAP=535.98 hasGaps=false
id=1 date = 20080718 00:46:30 open=535.59 high=535.73 low=535.58 close=535.7 volume=8 count=8 WAP=535.67 hasGaps=false
id=1 date = 20080718 00:47:00 open=535.24 high=535.95 low=535.24 close=535.95 volume=5 count=2 WAP=535.67 hasGaps=false
id=1 date = 20080718 00:47:30 open=535.37 high=535.72 low=535.28 close=535.54 volume=16 count=9 WAP=535.48 hasGaps=false
id=1 date = 20080718 00:48:00 open=535.52 high=536.16 low=535.47 close=535.55 volume=9 count=5 WAP=535.58 hasGaps=false
id=1 date = 20080718 00:48:30 open=535.49 high=535.78 low=535.41 close=535.47 volume=10 count=5 WAP=535.62 hasGaps=false
id=1 date = 20080718 00:49:00 open=535.55 high=535.85 low=535.2 close=535.85 volume=12 count=5 WAP=535.53 hasGaps=false
id=1 date = 20080718 00:49:30 open=534.91 high=535.47 low=534.91 close=535.47 volume=2 count=2 WAP=535.19 hasGaps=false
id=1 date = 20080718 00:50:00 open=535.27 high=535.62 low=535.25 close=535.62 volume=4 count=4 WAP=535.39 hasGaps=false
id=1 date = 20080718 00:50:30 open=534.99 high=534.99 low=534.47 close=534.47 volume=4 count=2 WAP=534.86 hasGaps=false
id=1 date = 20080718 00:51:00 open=534.68 high=534.88 low=534.43 close=534.74 volume=9 count=6 WAP=534.63 hasGaps=false
id=1 date = 20080718 00:51:30 open=534.6 high=535.05 low=534.09 close=534.09 volume=11 count=6 WAP=534.44 hasGaps=false
id=1 date = 20080718 00:52:00 open=534.13 high=534.21 low=533.16 close=533.89 volume=8 count=5 WAP=533.77 hasGaps=false
id=1 date = 20080718 00:52:30 open=533.68 high=534.09 low=533.68 close=533.9 volume=4 count=3 WAP=533.84 hasGaps=false
id=1 date = 20080718 00:53:00 open=534.39 high=534.39 low=533.56 close=533.89 volume=12 count=5 WAP=534.1 hasGaps=false
id=1 date = 20080718 00:53:30 open=533.96 high=534.35 low=533.36 close=534.02 volume=15 count=8 WAP=533.88 hasGaps=false
id=1 date = 20080718 00:54:00 open=533.9 high=534.02 low=533.37 close=533.5 volume=12 count=10 WAP=533.69 hasGaps=false
id=1 date = 20080718 00:54:30 open=532.96 high=533.58 low=532.96 close=533.34 volume=8 count=6 WAP=533.33 hasGaps=false
id=1 date = 20080718 00:55:00 open=533.69 high=533.71 low=533.69 close=533.71 volume=2 count=2 WAP=533.7 hasGaps=false
id=1 date = 20080718 00:55:30 open=533.78 high=534.16 low=533.61 close=533.71 volume=12 count=11 WAP=533.82 hasGaps=false
id=1 date = 20080718 00:56:00 open=534.08 high=534.29 low=534.0 close=534.29 volume=7 count=7 WAP=534.14 hasGaps=false
id=1 date = 20080718 00:56:30 open=533.88 high=534.81 low=533.82 close=534.81 volume=9 count=6 WAP=534.21 hasGaps=false
id=1 date = 20080718 00:57:00 open=534.31 high=535.73 low=534.31 close=535.73 volume=10 count=4 WAP=535.36 hasGaps=false
id=1 date = 20080718 00:57:30 open=535.96 high=536.26 low=535.96 close=536.26 volume=2 count=2 WAP=536.11 hasGaps=false
id=1 date = 20080718 00:58:00 open=535.67 high=535.67 low=535.67 close=535.67 volume=4 count=1 WAP=535.67 hasGaps=false
id=1 date = 20080718 00:58:30 open=536.05 high=536.05 low=536.01 close=536.01 volume=2 count=2 WAP=536.03 hasGaps=false


This was all run with the with the Trades option.
http://www.interactivebrokers.com/en/software/apiReleaseNotes/api9.php


Back to the Google Finance historical function part, I still haven't figured out the map from cid's to symbols, but I think I now have code that I can loop through to get some long term low-frequency stock data. I'd really like to figure out the mapping of contract id's to symbols. This, I think would be very helpful. But I'll do that tomorrow.

UpToNow

Up until now, I've been trying to figure out how exactly the IB Java API works. It's been an interesting process. I'm also starting to wonder the best way to incorporate it. Almost none of the functionality of normal sage is obviously asynchronous (if you don't count dsage). I'm wondering what's the best way to encorporate this type of model. To nutshell it, I'm wondering what parts of the API we actually need since the normal functionality of the IB API is seemingly not at all what we want... i.e. Implementing the EWrapper interface does not seem to be a reasonable goal (reasonably doable, just pointless to do the whole thing. - I don't think we're interested in getting a price tick every second, but maybe, I need to clarify with William.

Going back and forth between XML-RPC and Google Protocal Buffers seemed to be a waste of time because in the end, the GPB code genereated by protoc was not builidable. Potentially my fault, but even following the tutorial yielded code that didn't build. I also had real problems just compiling the examples.