29
edits
Telemetry/Custom analysis with spark: Difference between revisions
Telemetry/Custom analysis with spark (view source)
Revision as of 03:17, 19 November 2016
, 19 November 2016Long-running computation update
(→FAQ: Add sc.cancelAllJobs() tip) |
(Long-running computation update) |
||
| Line 171: | Line 171: | ||
3. Canceling execution of a notebook cell doesn't cancel any spark jobs that might be running in the background. If your spark commands seem to be hanging, try running `sc.cancelAllJobs()`. | 3. Canceling execution of a notebook cell doesn't cancel any spark jobs that might be running in the background. If your spark commands seem to be hanging, try running `sc.cancelAllJobs()`. | ||
=== How can I keep running after closing the notebook? === | |||
For long-running computation, it might be nice to close the notebook (and the ssh session) and look at the results later. Unfortunately, '''all cell output will be lost when a notebook is closed''' (for the running cell). To alleviate this, there are a few options: | |||
1. Have everything output to a variable. These values should still be available when you reconnect. | |||
2. Put %%capture at the beginning of the cell to store all output. [https://ipython.org/ipython-doc/3/interactive/magics.html#cellmagic-capture See the documentation]. | |||