Category Archives: Java

R vs Python for time series analysis

Working with R and Python simultaneously is a mess. Here is a short comparison how to do things with R and Python to analyze financial data.

Reading the data

library(curl)
library(xts)

url <- getURL("https://raw.githubusercontent.com/DmitryPukhov/stock-predictor/master/data/RI.RTSI_180101_180313.csv")
price.csv <- read.csv(textConnection(url))
price.csv <- price.csv[-c(1:2)]
price.ohlcv = as.xts(read.zoo(price.csv, index.column=c(1,2), tz="UTC", format="%Y%m%d%H:%M:%S"))
names(price.ohlcv) <- c("Open","High","Low","Close","Vol")

library(curl)

library(xts)

url <- getURL("https://raw.githubusercontent.com/DmitryPukhov/stock-predictor/master/data/RI.RTSI_180101_180313.csv")

price.csv <- read.csv(textConnection(url))

price.csv <- price.csv[-c(1:2)]

price.ohlcv = as.xts(read.zoo(price.csv, index.column=c(1,2), tz="UTC", format="%Y%m%d%H:%M:%S"))

names(price.ohlcv) <- c("Open","High","Low","Close","Vol")

Python

import pandas as pd

url <- getURL("https://raw.githubusercontent.com/DmitryPukhov/stock-predictor/master/data/RI.RTSI_180101_180313.csv")
price_ohlcv=pd.read_csv(url,
  parse_dates={'datetime':['<DATE>','<TIME>']}, 
  index_col='datetime', 
  usecols=['<DATE>','<TIME>','<OPEN>','<HIGH>','<LOW>','<CLOSE>','<VOL>'])
price_ohlcv.columns=['open','high','low','close','vol']

import pandas as pd

url <- getURL("https://raw.githubusercontent.com/DmitryPukhov/stock-predictor/master/data/RI.RTSI_180101_180313.csv")

price_ohlcv=pd.read_csv(url,

parse_dates={'datetime':['<DATE>','<TIME>']},

index_col='datetime',

usecols=['<DATE>','<TIME>','<OPEN>','<HIGH>','<LOW>','<CLOSE>','<VOL>'])

price_ohlcv.columns=['open','high','low','close','vol']

Reading financial data

library(quantmod)
sp500 = getSymbols('SP500', src='FRED', from='2017-05', to='2018-03')
sp500.tail()

library(quantmod)

sp500 = getSymbols('SP500', src='FRED', from='2017-05', to='2018-03')

sp500.tail()

Python

import pandas as pd
import numpy as np
from pandas_datareader.data import DataReader
sp500 = DataReader('SP500', data_source='fred', start='2016-05', end='2018')
sp500.tail()

import pandas as pd

import numpy as np

from pandas_datareader.data import DataReader

sp500 = DataReader('SP500', data_source='fred', start='2016-05', end='2018')

sp500.tail()

View table

head(price.ohlcv)
tail(price.ohlcv)
str(price.ohlcv)
summary(price.ohlcv)

head(price.ohlcv)

tail(price.ohlcv)

str(price.ohlcv)

summary(price.ohlcv)

Python

price.head()
price.tail()
price.info()
price.describe()

price.head()

price.tail()

price.info()

price.describe()

Selecting time series

sp500.winter <- SP500['2017-12/2018-02']
sp500.monthly <- to.period(SP500,on='month')
sp500.monthly.mean <- apply.monthly(SP500, mean)

sp500.winter <- SP500['2017-12/2018-02']

sp500.monthly <- to.period(SP500,on='month')

sp500.monthly.mean <- apply.monthly(SP500, mean)

Python

sp500_winter = SP500['2017-01':'2017-02']
sp500_monthly = SP500.asfreq('M')
sp500_monthly_mean = sp500.resample('M').mean()

sp500_winter = SP500['2017-01':'2017-02']

sp500_monthly = SP500.asfreq('M')

sp500_monthly_mean = sp500.resample('M').mean()

Plotting

plot(sp500)

1	plot(sp500)

Python

import matplotlib.pyplot as plt
SP500.plot()
plt.show()

import matplotlib.pyplot as plt

SP500.plot()

plt.show()

Train model

library(astsa)
n = nrow(price.cl)
# train/test split
price.cl.train <- price.cl[1:round(0.7 * n)]
price.cl.test <- price.cl[(round(0.7 * n)+1):n]

# Predict
price.cl.pred.raw <- sarima.for(price.cl, p = 1, d = 1, q = 1, P = 0, D = 0, Q = 3, S = 12, n.ahead=9)
price.cl.pred <- xts(price.cl.pred.raw$pred, timeBasedSeq("20180220/20180313"))

# Draw the plot
plot(merge(price.cl, price.cl.pred), type="n")
points(price.cl.train, col = "blue")
points(price.cl.pred, col = "red", lty=2)

library(astsa)

n = nrow(price.cl)

# train/test split

price.cl.train <- price.cl[1:round(0.7 * n)]

price.cl.test <- price.cl[(round(0.7 * n)+1):n]

# Predict

price.cl.pred.raw <- sarima.for(price.cl, p = 1, d = 1, q = 1, P = 0, D = 0, Q = 3, S = 12, n.ahead=9)

price.cl.pred <- xts(price.cl.pred.raw$pred, timeBasedSeq("20180220/20180313"))

# Draw the plot

plot(merge(price.cl, price.cl.pred), type="n")

points(price.cl.train, col = "blue")

points(price.cl.pred, col = "red", lty=2)

.. to be continue

Find jar with class

Leave a reply

This helped me to resolve java.lang.ClassNotFoundException. Bash command scans jar files in the folder and prints who contains given class.

find . -name '*.jar' -print0 |  xargs -0 -I '{}' sh -c 'jar tf {} | grep Hello.class &&  echo {}'

1	find . -name '*.jar' -print0 \| xargs -0 -I '{}' sh -c 'jar tf {} \| grep Hello.class && echo {}'

Maven: copy dependencies

Leave a reply

Getting dependent jars together in a folder. Can be used as an alternative to fat jar.

mvn dependency:copy-dependencies -DoutputDirectory=${project.build.directory}/lib

1	mvn dependency:copy-dependencies -DoutputDirectory=${project.build.directory}/lib

Continue reading →

Git server on Windows

Leave a reply

Spent some time struggling with Git server on Windows (!). Memorize it here to not forget.

Init Windows git server

On Windows. git server create a folder for project, let it be c:/GitRepos/myProject.git Enter that directory with cygwin and run command there:

git init --bare

1	git init --bare

Access from client.

Note: Windows user git includes server name with plus sign: server1+git

git clone server1+git@server1:c:/GitRepos/myProject.git
echo "Test commit" > test.txt
git commit -m "Test commit"
git push origin master

git clone server1+git@server1:c:/GitRepos/myProject.git

echo "Test commit" > test.txt

git commit -m "Test commit"

git push origin master

Kill Linux process

Leave a reply

Filter processes by string pattern (let it be “java” processes):

ps -eaf | grep java

1	ps -eaf \| grep java

Find my process in resulted list and kill by id:

kill -9 123

1	kill -9 123

Kill all processes by “java” pattern:

kill -9 `ps -eaf | grep java | grep -v grep | awk '{print$2}'`

1	kill -9 `ps -eaf \| grep java \| grep -v grep \| awk '{print$2}'`

Maven offline mode: do not check external repositories

Leave a reply

How to package the project with mvn command when external repos are not available, but we are happy to get dependencies from local repositories:

mvn clean -o package

1	mvn clean -o package

Cancel postgres process

Leave a reply

Show list of active processes to see pid there

SELECT query, * FROM pg_stat_activity where state = 'active'

1	SELECT query, * FROM pg_stat_activity where state = 'active'

Cancel the process by pid

select pg_cancel_backend(<pid>)

1	select pg_cancel_backend(<pid>)

Apache Spark Feature http://apache.org/xml/features/xinclude is not recognized

Leave a reply

Problem: Apache Spark 1.3.1 application produces the following error:

Exception in thread "main"java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.

1	Exception in thread "main"java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.

Fix: edit pom.xml to use older version of xercesImpl Continue reading →

Maven: skip tests and others

Leave a reply

Some Maven commands I often use recently:

Skip tests (not for deployment to production :))

mvn clean package -Dmaven.test.skip=true

1	mvn clean package -Dmaven.test.skip=true

Continue reading →

Signing jar manually

Leave a reply

Steps to manually sign a jar

Generate a new key with alias dmitry-dev

keytool -genkey -v -keystore ~/dmitry-dev.keystore -alias dmitry-dev -keyalg RSA -keysize 2048 -validity 10000

1	keytool -genkey -v -keystore ~/dmitry-dev.keystore -alias dmitry-dev -keyalg RSA -keysize 2048 -validity 10000

Enter password and passphrase, remember them Continue reading →

Dmitry Pukhov

Software Developer, Architect

Category Archives: Java

R vs Python for time series analysis

Reading the data

Reading financial data

View table

Selecting time series

Plotting

Train model

Find jar with class

Maven: copy dependencies

Git server on Windows

Init Windows git server

Access from client.

Kill Linux process

Maven offline mode: do not check external repositories

Cancel postgres process

Apache Spark Feature http://apache.org/xml/features/xinclude is not recognized

Maven: skip tests and others

Signing jar manually

Generate a new key with alias dmitry-dev