Useful links
- nightly and aurora
- crashes by build (for nightly and aurora)
- beta
- 47.0b1 crashes (only in first 2 minutes after installing) (only on bug 772330 AMD CPUs) (e10s test cohort) (e10s control cohort)
- 47.0b2 crashes (only in first 2 minutes after installing) (only on bug 772330 AMD CPUs) (e10s test cohort) (e10s control cohort)
- 47.0b3 crashes (only in first 2 minutes after installing) (only on bug 772330 AMD CPUs) (e10s test cohort) (e10s control cohort)
- 47.0b4 crashes (only in first 2 minutes after installing) (only on bug 772330 AMD CPUs) (e10s test cohort) (e10s control cohort)
- 47.0b5 crashes (only in first 2 minutes after installing) (only on bug 772330 AMD CPUs) (e10s test cohort) (e10s control cohort)
- 47.0b6 crashes (only in first 2 minutes after installing) (only on bug 772330 AMD CPUs) (e10s test cohort) (e10s control cohort)
- 47.0b7 crashes (only in first 2 minutes after installing) (only on bug 772330 AMD CPUs) (e10s test cohort) (e10s control cohort)
- 47.0b8 crashes (only in first 2 minutes after installing) (only on bug 772330 AMD CPUs) (e10s test cohort) (e10s control cohort)
- release
- general
- A rough guide to Nightly crash analysis.
2016-05-27 - dbaron
2016-05-26 - mccr8
2016-05-25 Nightlies
- Top crash was mozilla::net::nsStreamListenerWrapper::OnStopRequest (16 crashes), but from a single installation, and it doesn't seem to have happened anywhere else in the last 7 days, so I didn't file anything. Alright, I went back and filed bug 1276080 for this crash, as it looks avoidable.
- Filed bug 1276062 - Crash in mozilla::layers::BasicLayerManager::PushGroupForLayer about an intentional crash.
2016-05-25 - jchen
2016-05-24 Nightlies
- Windows #6: free_impl | nsACString_internal::Replace
- Filed bug 1275811
- Windows #7: nsAString_internal::Assign | nsString::nsString
- Filed bug 1275809
- Windows #8: base::win::PEImage::GetProcOrdinal
- Filed bug 1275813
- Android #10: libart.so@0x2f1b62
- Classified as bug 1271103
2016-05-25 - marcia
- Filed bug 1275538 for a Media crash
2016-05-25 - kanru
I looked at May 23 Nightly on Windows
- Windows
- #8 [@ js::fun_apply] 7 crashes, probably from single installation
- Filed bug 1275495
- #11 [@ js::jit::CodeGenerator::visitOutOfLineTypeOfV ]
- Filed bug 1275500
- #13 [@ shutdownhang | ntdll.dll@0x906fa] 6 crashes
- 5 are in mozilla::layers::CompositorThreadHolder::Shutdown
- Filed bug 1275505
- 1 is in mozilla::net::nsHttpConnectionMgr::Shutdown, didn't file
- 5 are in mozilla::layers::CompositorThreadHolder::Shutdown
- #16 [@ @0x0 | nsJSCID::GetService] 4 crashes from single installation, didn't file
- #47 [@ nsTHashtable<T>::GetEntry | nsInterfaceHashtable<T>::Get | mozilla::dom::indexedDB::`anonymous namespace::DatabaseConnection::GetCachedStatement ], 9 crashes on Nightly since May 11.
- Filed bug 1275518
- #8 [@ js::fun_apply] 7 crashes, probably from single installation
2016-05-24 - ting
05-22 Nightly
- Windows
- #1 [shutdownhang | ntdll.dll@0xa5164] 27 crashes
- 21 are bug 1268559
- 1 is bug 1267752, thread 58 is with unresolved symbols in msmpeg2vdec.dll
- 1 is bug 1274399, marked bug 1257908 a duplicate of bug 1274399
- 1 is bug 1158189, ni'd :mcmanus
- 1 is bug 1273475
- Filed bug 1275167 for hang in nsEventQueue::GetEvent
- Filed bug 1275162 for hang in CacheIOThread::Shutdown
- #3 [shutdownhang | ntdll.dll@0x4bb7a] 8 crashes
- 6 are bug 1268559, 1 is bug 1158189, 1 is bug 1275167
- #7 [nss3.dll@0x1e9700 | kernelbase.dll@0x98b5] is bug 1229252
- #8 [shutdownhang | ntdll.dll@0x906fa] 6 crashes
- 2 are bug 1268559, 2 are bug 1158189, 2 are bug 1275167
- #11 [xul.dll@0x245d978 | mozilla::ipc::DeserializeInputStream] 5 crashes from a single installation, ignored
- #12 [MustSkipMarking<T>] 4 crashes from 2 installations, filed bug 1275172
- #19 [RtlpCoalesceFreeBlocks | RtlpFreeHeap | RtlpCoalesceFreeBlocks | HeapFree | atiumdag.dll@0x4588c] 3 crashes from the same installation, ignored
- #20 [js::LookupOwnPropertyPure] 3 crashes from 2 installations, filed bug 1275185
- #21 [mozilla::layers::CompositorD3D9::DrawQuad] filed bug 1275179
- #42 [nsPrintData::DoOnStatusChange] filed bug 1275194, ni'd :bobowen
- #1 [shutdownhang | ntdll.dll@0xa5164] 27 crashes
2016-05-23 - njn
I looked at May 20 & 21 builds on Windows, Mac, Linux. Something is wrong with the Android results, and the searched linked to from dbaron's page for May 20, 21, 22 are giving zero result. (Turns out bug 1274488 is the cause.)
- Filed bug 1274888 for Crash in mozilla::EventStateManager::GetContentViewer and fixed it.
- Filed bug 1274892 for mozilla::MediaSourceDemuxer::GetTrackDemuxer. jya immediately fixed it.
- Filed bug 1274882 for nvwgf2umx.dll@0x7bdb85 and other crashes in that DLL.
- Filed bug 1274886 for IPCError-content | (msgtype=0x800017,name=???) Route error: message sent to unknown actor ID.
- Filed bug 1274887 for Fold.
- Filed bug 1274895 for js::GlobalHelperThreadState::ionLazyLinkListRemove.
- Filed bug 1274898 for Crash in libgtk-3.so.0.2000.5@0x1b9bd1.
- Ignored some single-installation crashes.
2016-05-20 - dbaron
- 05-20 Windows
- ReportCorruption crashes are all one user; ignored
2016-05-20 - mccr8
05-18 Nightly
- Top crash is bug 1271102, a release assert for large messages. I filed bug 1274404 for adding annotations to make it easier to categorize these. I also started working on bug 1272423 to make a common source of these less frequent.
- Filed #5 bug 1274619 - Null deref crash in js::jit::ICGetProp_Primitive::Compiler::getStub, all from three different close install times.
- Filed #7 bug 1274625 - Crash in js::jit::JitCode::copyFrom, all from a single install time.
- Filed #8 bug 1274621 - Crash in nsCSSScanner::ScanHash, all from a single install time.
- Filed #11 bug 1274630 - Stack buffer overrun in guard64.dll@0x494b. Maybe we should blocklist this.
2016-05-18 - jchen
05-17 Nightly
- Looked at Windows #10 mozalloc_abort | NS_DebugBreak | mozilla::ipc::FatalError | mozilla::dom::PBrowserChild::FatalError | mozilla::dom::PBrowserChild::Read
- Only 2 installations. Going to wait and see.
- Many Windows signatures without symbols
- Hopefully fixed by bug 1272278 in next Nightly.
- No new signatures for for Mac, Linux, Android
2016-05-18 - kanru
I looked at the May 16 build. Symbols are still missing for 64bit windows for this build.
- Connected [@ nss3.dll@0x1e9700] to bug 1229252
- Looking at [@ mozalloc_abort | xul.dll@0x77f718] looks like [@ mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | NS_InvokeByIndex]
- nothing interesting so far in Mac, Linux, or Android
2016-05-18 - ting
For 5-15 Nightly.
- Windows
- #3 [igdumd32.dll@0x424de] 22 crashes from 1 installation, ignored.
- #4 [xul.dll@0x1739c82 | strspn] crashed in MediaFormatReader::RequestAudioData(), connected to bug 1273018 which just landed on 5/17.
- #6 [mozalloc_abort | xul.dll@0x7c7428 | xul.dll@0xb905b] crashed from JIT calling nsDebugImpl::Abort(), could be bug 1187196 but filed bug 1273770.
- #7 [xul.dll@0x31b103] crashed in nsStyleMargin::GetMarginNoPercentage, connected to bug 1273360.
- #9 [xul.dll@0x1885e8f] 8 crashes from 1 installation, ignored.
- #10 [xul.dll@0x1739c82 | ucrtbase.dll@0xe41bf] connected to bug 1273018.
- #11 [xul.dll@0xfd40a6 | SystemTimeToFileTime] 6 crashes from 1 installation, ignored.
- #13 [xul.dll@0xccfa63 | strspn] connected to bug 1271102.
- #15 [PL_HashTableAdd | nsLocale::AddCategory] 4 crashes from 1 installation, ignored.
- #19 [moz_abort | arena_run_split | arena_malloc_large | malloc_impl | xul.dll@0x1e25b2] connected to bug 1267924, added comments.
- Mac
- #4 [shutdownhang | js::] started crashing from 20160507030302, filed bug 1273792.
- Android
- #5 [@0xf490173a] 5 crashes from 1 installation, ignored.
2016-05-16 - njn
I looked at May 13 and 14 builds.
- bug 1272738 badly affects usefulness of Windows crash reports and greatly reduced my ability to analyze these builds' crashes. I added extra data, a needinfo and a tracking49 to that bug.
- Filed bug 1273015 for sr2osd.dll@0x438e.
- Filed bug 1273018 for Crash in mozilla::MediaFormatReader::UpdateReceivedNewData.
- Filed bug 1273020 for mozilla::a11y::ApplicationAccessibleWrap::Name and posted a patch to fix it.
- Filed bug 1273043 for mozilla::net::HttpChannelChild::DoOnStartRequest and posted a patch that might fix it.
- Filed bug 1273048 for mozilla::PseudoElementForStyleContext.
2016-05-13 - dbaron
- Windows 5-12 (Thursday) builds:
- seems like something wrong with the symbols for the nightly (bug 1272738)
- "xul.dll@0xabeeb0 | PR_NativeRunThread | pr_root" is a set of shutdownhang crashes
- seems like something wrong with the symbols for the nightly (bug 1272738)
- Windows, May 12 (again)
- connected bug 1272746 and poked for tracking
- filed bug 1272815
- connected gfxWindowsPlatform::UpdateBackendPrefs crashes to bug 1116540
- connected new crashes to bug 1266209
- Mac, May 12
- connected Mac signature to bug 1272746
- Linux, May 12
- connected Linux signature to bug 1272746
- Android, May 12
- filed bug 1272817
- Windows, May 13: no data yet since 64-bit builds have bad bug 1272738, and 32-bit builds were built very late and hadn't reported crashes yet
- nothing interesting so far in May 13 Mac, Linux, or Android
2016-05-13 - mccr8
For 5-11 Nightly.
- #2 Windows crash is [@ shutdownhang | ntdll.dll@0xa5164], which is missing symbols. I posted in the Windows missing symbols bug (bug 1270190).
- Two signatures with xul.dll, which doesn't seem like it should happen. I filed bug 1272738 for that.
- bug 1272750 - Filed a bug for the #8 Windows crash, in NSS, though it appears to be all one user.
- bug 1271102 - Added a signature for OSX for this message size assertion, which is the #1 crash on OSX.
2016-05-11 - jchen
For 05-10 Nightly.
- Revisited Windows #1 crash nsXPCWrappedJS::CallMethod (bug 1271019)
- Commented on addon correlation and regression range
- Revisited Windows #5 crash shutdownhang | ntdll.dll@0x4bb7a, #6 crash shutdownhang | ntdll.dll@0xa5164, and #8 crash shutdownhang | ntdll.dll@0x4d3fa
- Many but not all seem input related (CThreadInputMgr and/or msctf.dll in stack)
- Maybe we should file a catch-all "ntdll.dll@<hex address>" bug?
- Revisited Windows #9 crash msmpeg2vdec.dll@0x25a3fb (bug 1180561)
- Nagged assignee
2016-05-11 - kanru
I looked at May 9.
- windows nightly #6 [@ shutdownhang | ntdll.dll@0x4bb7a ] mostly bug 1268559
- 4 reports from 2 installations are shutdownhang in mozilla::net::nsHttpConnectionMgr::Shutdown, didn't file a bug.
- windows nightly #11 [@ mozalloc_abort | NS_DebugBreak | IPC::ParamTraits<T>::Write]
- Filed bug 1271888 Crash in mozalloc_abort | NS_DebugBreak | IPC::ParamTraits<mozilla::net::NetAddr>::Write "Unknown socket family"
- Filed bug 1271887 to separate reports of IPC::ParamTraits<T> to the specific types
- windows nightly #14
- Filed bug 1271890 Crash in base::win::PEImage::GetProcAddress
- android nightly #2 std::rethrow_exception in mozilla::gl::CreateSurfaceForWindow
- Added to bug 1271103
2016-05-09 - njn
I looked at May 8.
- Filed bug 1271522 for nsIFrame::TrackingVisibility.
- Filed bug 1271525 for igdumd64.dll@0xd0ef0.
- Filed bug 1271527 for mozilla::net::WebSocketChannel::Notify.
- Filed bug 1271529 about getting better crash signature for Rust panics. Not sure exactly how the regex matching will work on some of the stack frames.
- Minor updates to a couple of other bugs.
2016-05-09 - njn
I looked at May 6 and 7. Android's a mess at the moment.
- Filed bug 1271164 and uploaded a fix, for gfxUserFontSet::UserFontCache::Entry::ReportMemory.
- Filed bug 1271163 for atidxx64.dll@0x1dd994.
- Filed bug 1271174 for java.lang.NullPointerException: at org.mozilla.gecko.telemetry.UploadTelemetryCorePingCallback$1.run(UploadTelemetryCorePingCallback.java).
- File bug 1271175 for java.lang.IllegalArgumentException: The key must be an application-specific resource id. at android.view.View.setTag(View.java).
- Added some data to a few others, and added tracking49? to a couple.
- Added other crash signatures to bug 1271103.
2016-05-06 - dbaron
I looked at builds for May 5, and a little bit of aurora for May 6.
- filed bug 1270963 on new Android topcrash
- poked various other bugs to try to make them progress
- filed bug 1271019 on May 6 topcrash (XPCWrappedJS MOZ_CRASH())
- filed bug 1271020 on Android May 6 topcrash
2016-05-05 - mccr8
I looked at builds for 05-04
- #1 bug 1270652 - Crash in mozilla::layers::FindScrolledLayerForScrollbar. 17 crashes, but it looks like it is a single user. I filed a bug anyways.
- #7 [@ shutdownhang | ntdll.dll@0x4bb7a ] - This crash has no bug on file any more, but at least some of it is bug 1268559. Due to bug 1270190, the crash signature is crummy and a bunch of random issues are just bucketed together.
2016-05-04 - jchen
- windows nightly #5: shutdownhang | ntdll.dll@0x4bb7a
- Several reports have "CompositorBridgeParent::ShutDown" and/or "CThreadInputMgr::PeekMessageW" in the stack, but I'm not positive it's related to bug 1268559.
- windows nightly #6: PLDHashTable::Remove | mozilla::a11y::DocAccessible::UnbindFromDocument
- Added signature to bug 1268320.
- windows nightly #7: wbload.dll@0x23c7c
- Filed bug 1270345.
- windows nightly #9: shutdownhang | WaitForSingleObjectEx | CThreadInputMgr::PeekMessageW
- Related to shutdownhang | ntdll.dll@0x4bb7a, but again, not sure of relation to bug 1268559.
- android aurora #5: Crash in java.lang.NullPointerException: Attempt to invoke virtual method int android.graphics.Bitmap.getWidth() on a null object reference at org.mozilla.gecko.favicons.LoadFaviconTask.doInBackground(LoadFaviconTask.java)
- Filed bug 1270348
2016-05-03 - njn
I looked at builds for May 01 and 02.
Windows, May 01:
- #1, #2, #3: shutdownhang | ...: Noted again in bug 1268920 that this is a bad one.
- #4: PLDHashTable::Remove | mozilla::a11y::EventTree::Process: Noted in bug 1268320 that this is a top crash and requested tracking for 49.
- #5: libovrrt32_1.dll@0xc65f: Added more info in bug 1267150.
- #7: nsACString_internal::Assign | nsACString_internal::Assign | mozilla::gmp::GetContentParentFromDone::Done: bug 1268714 appears to have fixed this one; no sign of it in the May 02 Nightly.
- #9: IsAboutToBeFinalizedInternal<T>: ignored, difficult JS GC one with varying stacks
- #11: mozilla::layers::SyncObjectD3D11::FinalizeFrame: added info to the bug.
- #13: d3d10warp.dll@0x6d412: filed bug 1270000
- #16: moz_abort | arena_run_split | arena_malloc_large | malloc_impl | PLDHashTable::Add | CCGraphBuilder::NoteRoot: ignored; abort within jemalloc, hard to act on
Windows, May 02:
- #1: mozilla::FrameLayerBuilder::PaintItems: ignored; 48 crashes from one installation, and no sign of it in the May 03 crash reports.
- #6: mozilla::AudioBlockCopyChannelWithScale_SSE: reopened bug 1266302.
- #13: mozilla::places::`anonymous namespace::FetchPageInfo: ignored; six crashes from one installation.
- #14: NS_InitXPCOM2: ignored; five crashes from one installation.
- #19: winmm_stream_init: filed bug 1270004.
Mac, May 01 and 02:
- Nothing of note.
Linux, May 01 and 02:
- Nothing of note.
Android, May 01:
- #1: ElfLoader::~ElfLoader: added data to bug 1164027
Android, May 02:
- #3, #6: libstagefright ones: couldn't make sense of these. All from a single installation, and the stacks are really weird.
2016-05-02 - njn
Note that Skia backends have been enabled (I think?) on some platforms recently, so it's worth keeping an eye out for Skia-related crashes, which usually involved functions with a "Sk" prefix.
I looked at builds for 4/29 and 4/30.
Windows 04/29:
- #1, #2: Noted that bug 1268920 is the #1 and #2 topcrash.
- #3: Interpret: ignored; 18 crashes from a single installation.
- #6: mozalloc_abort | NS_DebugBreak | mozilla::ipc::FatalError | mozilla::net::PHttpChannelChild::FatalError | mozilla::net::PHttpChannelChild::OnMessageReceived: matches a supposedly fixed bug. 12 crashes from one installation. I needinfo'd kanru.
- #10: mswsock.dll@0x355e: ignored; 6 crashes from a single installation, stack trace has DLL weirdness in it.
- #12: mozilla::gfx::Path::TransformedCopyToBuilder: added new diagnostic info from today's crashes to bug 1266209.
- #13: shutdownhang | ntdll.dll@0x2bea: ignored; 5 hangs from a single installation.
- #14: xfelkn.dll@0xba3ba: ignored; 5 hangs from one installation. Google says almost nothing about this DLL.
Windows 04/29:
- #5: js::jit::JitCode::copyFrom: ignored; 5 crashes from one installation.
- #6: mozilla::dom::WorkerGlobalScopeBinding_workers::genericGetter: ignored; 5 crashes from one installation.
- #7: NS_strdup | PREF_CopyCharPref: ignored; 4 crashes from one installation
- #13: d3dcompiler_47.dll@0x23aacd: ignored; 3 crashes from one installation, unhelpful stack traces
- #16: nhasusstrixosd.dll@0x663e: filed bug 1269244.
Mac 04/29, 04/30:
- Nothing of note.
Linux 04/29:
- #2: mozilla::IncrementalFinalizeRunnable::ReleaseNow: filed bug 1269245.
- #5: MaiAtkObject::Shutdown: reopened bug 1263517 in light of reoccurrences.
Linux 04/30:
- Nothing of note.
Android 04/29:
- #1: java.lang.NullPointerException: at android.widget.AdapterView.performItemClick(AdapterView.java): ignored; 5 crashes from 1 installation.
Android 04/30:
- #1: SkDraw::ShouldDrawTextAsPaths : filed bug 1269247.
2016-04-29 - dbaron
looked at builds for 4/28 and some of aurora for 4/29 as well
- filed bug 1268920 on @0x0 | js::Proxy::get crashes on aurora only
- looked into ParseSingleTransform crash briefly, blamed bad hardware
- filed bug 1269013 on Fennec stack overflow crashes in _cairo_spline_decompose_into
- filed bug 1269028 on bizarre and probably unactionable floorf crashes
- filed bug 1269036 on IPC aborts
- and bug 1269033 on improving the signature with prefix_signature_re
- Aurora PREF_GetIntPref crashes were one user; skipped
- debugged an instance of bug 1267647
2016-04-28 - mccr8
I looked at builds for 4/27.
- bug 1268559: I put a bunch of the shutdown hangs in here. They are mostly happening during compositor shutdown, and that code got a big revamp last week, so maybe that's to blame. It is just spinning the event loop so maybe it isn't that code's fault per se.
- bug 1268714: Near-null crash in gmp::GetContentParentFromDone::Done. Filed this. Looks like a new regression.
- bug 1268719: Very large allocations in refill_callback_duplex. Around 1GB. Looks like a regression.
- bug 1268721: Null deref crash in GetSurfaceData() called from PuppetWidget::SetCursor(). This looks like an old crash, but it looks simple to fix.
2016-04-27 - jchen
- OOM | unknown | js::AutoEnterOOMUnsafeRegion::crash | JS::Zone::getUniqueIdInfallible
- Need to reassess; low volume OOM
- PLDHashTable::Remove | mozilla::a11y::EventTree::Process
- Filed bug 1268320; code in EventTree.cpp appears to be recently changed
- base::win::PEImage::GetProcAddress
- Need to reassess; not a recent crash and is inside imported code
- libgobject-2.0.so.0.4400.1@0x33af4
- Filed bug 1268324; appears to have started after bug 1262563 landed
- Ignored single install crashes
- mozilla::css::GroupRule::~GroupRule
- nhasusstrixosd.dll@0x663e
- ld-2.23.so@0xb571 (single install in 48.0a2, couple of installs in 47.0a2)
- Ignored shutdown hangs
- shutdownhang | WaitForSingleObjectEx | PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent
- shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent
- shutdownhang | ntdll.dll@0x4bb7a
- shutdownhang | ntdll.dll@0x9386a