Telemetry/Custom analysis with spark: Difference between revisions

Telemetry/Custom analysis with spark (view source)

191 bytes added , 15 November 2016

→‎FAQ: Add sc.cancelAllJobs() tip

13

edits

Revision as of 17:45, 20 October 2016 (view source) Fbertsch (talk \| contribs) (→‎Using Spark: Add spark UI) ← Older edit		Revision as of 19:45, 15 November 2016 (view source) Sunahsuh (talk \| contribs) (→‎FAQ: Add sc.cancelAllJobs() tip) Newer edit →
Line 169:		Line 169:

	2. The connection from PySpark to the Spark driver might be lost. Unfortunately the best way to recover from this for the moment seems to be spinning up a new cluster.		2. The connection from PySpark to the Spark driver might be lost. Unfortunately the best way to recover from this for the moment seems to be spinning up a new cluster.

			3. Canceling execution of a notebook cell doesn't cancel any spark jobs that might be running in the background. If your spark commands seem to be hanging, try running `sc.cancelAllJobs()`.