Bigdata ideas

I have study purpose Cassandra on Windows 7 (3 keyspaces) and want to copy it on new Cassandra cluster (Linux) 1) Import keyspaces, just structures. cqlsh -e "DESCRIBE MTS_BARS" > C:\cassdump\mts_bars.cql cqlsh -e "DESCRIBE MTS_META" > C:\cassdump\mts_meta.cql cqlsh -e "DESCRIBE MTS_SRC" > C:\cassdump\mts_src.cql There are some files that can be opened with notepad and contains SQL like queries to CREATE KEYSPACE, CREATE TABLE. 2) Export data with COPY TO ( simple-data-importing-and-exporting ) Connect to cqlsh and execute commands: COPY MTS_SRC.TICKS TO 'C:\cassdump\mts_src_ticks.cql' WITH HEADER=FALSE; COPY MTS_BARS.BARS TO 'C:\cassdump\mts_bars_bars.cql' WITH HEADER=FALSE; COPY MTS_BARS.bars_future TO 'C:\cassdump\mts_bars_bars_future.cql' WITH HEADER=FALSE; COPY MTS_BARS.bars_test TO 'C:\cassdump\mts_bars_test.cql' WITH HEADER=FALSE; COPY MTS_BARS.LASTBARS TO 'C:\cassdump\mts_bars_lastbars.c...

I have study Spark cluster and Cassandra cluster that described in  this topic . Here I am going to work (make some examples) with Spark cluster under spark-shell and with data from Cassandra. First of all, we need to copy Cassandra driver into $SPARK_HOME/jars (/opt/spark-2.3.2/jars/) 1) Read the whole table from Cassandra into Dataset[org.apache.spark.sql.Row] [root@smn conf]# spark-shell --jars /opt/spark-2.3.2/jars/spark-cassandra-connector-assembly-2.3.2.jar --driver-memory 3g --executor-memory 3g --conf spark.cassandra.connection.host=192.168.122.192 2018-10-12 06:51:51 WARN Utils:66 - Your hostname, smn resolves to a loopback address: 127.0.0.1; using 192.168.122.219 instead (on interface eth0) 2018-10-12 06:51:51 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 2018-10-12 06:51:52 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log leve...