Log from creating calibs for DM-8913 ==================================== price@master:~ $ for dd in 2016-01-15/ 2016-02-04/ 2016-03-08/ 2016-04-12/ 2016-06-03/ 2016-06-28/ 2016-08-04/ 2016-08-25/ ; do python getVisits.py /lustre2/Subaru/SSP/CALIB_20151201/BIAS/$dd/NONE/*/BIAS-049.fits; done price@master:~ $ for dd in 2016-01-14/ 2016-02-04/ 2016-03-17/ 2016-04-13/ 2016-06-05/ 2016-06-28/ 2016-07-30/ 2016-08-25/ ; do python getVisits.py /lustre2/Subaru/SSP/CALIB_20151201/DARK/$dd/NONE/*/DARK-049.fits; done price@master:~ $ for dd in 2015-11-19/HSC-Z 2016-01-05/HSC-R 2016-01-07/HSC-Z 2016-01-08/HSC-G 2016-01-09/HSC-Y 2016-01-10/HSC-I 2016-01-11/NB0921 2016-02-03/HSC-Y 2016-02-04/HSC-R 2016-02-07/HSC-I2 2016-02-08/NB0816 2016-02-11/HSC-G 2016-03-04/HSC-I2 2016-03-07/HSC-G 2016-03-09/HSC-R 2016-03-12/HSC-Z 2016-03-14/HSC-Y 2016-04-01/HSC-R 2016-04-03/HSC-I2 2016-04-04/HSC-G 2016-04-09/NB0921 2016-04-15/HSC-Y 2016-06-01/HSC-Z 2016-06-02/HSC-G 2016-06-05/HSC-R 2016-06-07/HSC-I2 2016-06-12/HSC-Y 2016-06-28/HSC-Y 2016-06-29/HSC-I2 2016-06-30/HSC-R 2016-07-06/HSC-Z 2016-07-29/HSC-Z 2016-07-30/HSC-G 2016-07-31/HSC-Y 2016-08-02/NB0921 2016-08-06/HSC-I2 2016-08-06/HSC-R2 2016-08-25/HSC-R2 2016-08-28/HSC-I2 2016-08-29/HSC-G 2016-09-03/NB0387; do python getVisits.py /lustre2/Subaru/SSP/CALIB_20151201/FLAT/$dd/*/FLAT-049.fits; done price@master:~ $ for dd in 2015-11-12/ 2016-01-09/ 2016-01-11/ 2016-02-03/ 2016-03-14/ 2016-03-15/ 2016-04-07/ 2016-04-08/ 2016-04-09/ 2016-04-11/ 2016-04-14/ 2016-04-15/ ; do python getVisits.py /lustre2/Subaru/SSP/CALIB_20151201/FRINGE/$dd/*/*/FRINGE-049.fits; done Bias: 2016-01-15 53396..53414:2 2016-02-04 55122..55140:2 2016-03-08 59760..59778:2 2016-04-12 68534..68552:2 2016-06-03 71216..71234:2 2016-06-28 74748..74766:2 2016-08-04 81994..82012:2 2016-08-25 83820..83838:2 Dark: 2016-01-14 53030..53038:2 2016-02-04 55112..55120:2 2016-03-17 63352..63360:2 2016-04-13 68948..68956:2 2016-06-05 71212^71214^72206..72210:2 2016-06-28 74738..74746:2 2016-07-30 80554..80562:2 2016-08-25 83840..83848:2 Flat: 2015-11-19/HSC-Z 48002..48020:2 2016-01-05/HSC-R 48906..48924:2 2016-01-07/HSC-Z 50064..50082:2 2016-01-08/HSC-G 50720..50738:2 2016-01-09/HSC-Y 50980..50998:2 2016-01-10/HSC-I 51388..51406:2 2016-01-11/NB0921 51740..51758:2 2016-02-03/HSC-Y 53782..53798:2 2016-02-04/HSC-R 54158..54176:2 2016-02-07/HSC-I2 56044..56062:2 2016-02-08/NB0816 56694..56712:2 2016-02-11/HSC-G 57300..57318:2 2016-03-04/HSC-I2 58080..58098:2 2016-03-07/HSC-G 59054..59072:2 2016-03-09/HSC-R 59798..59816:2 2016-03-12/HSC-Z 61134..61152:2 2016-03-14/HSC-Y 62630..62648:2 2016-04-01/HSC-R 64430..64448:2 2016-04-03/HSC-I2 65756..65774:2 2016-04-04/HSC-G 66578..66596:2 2016-04-09/NB0921 67746..67764:2 2016-04-15/HSC-Y 69268..69286:2 2016-06-01/HSC-Z 69624..69642:2 2016-06-02/HSC-G 69870..69888:2 2016-06-05/HSC-R 71554..71572:2 2016-06-07/HSC-I2 72240..72258:2 2016-06-12/HSC-Y 74008..74026:2 2016-06-28/HSC-Y 74490..74508:2 2016-06-29/HSC-I2 74786..74804:2 2016-06-30/HSC-R 75024..75042:2 2016-07-06/HSC-Z 77406..77424:2 2016-07-29/HSC-Z 79946..79964:2 2016-07-30/HSC-G 80270..80288:2 2016-07-31/HSC-Y 80590..80608:2 2016-08-02/NB0921 81202..81220:2 2016-08-06/HSC-I2 82600..82618:2 2016-08-06/HSC-R2 82364..82382:2 2016-08-25/HSC-R2 83750..83768:2 2016-08-28/HSC-I2 84326..84344:2 2016-08-29/HSC-G 84574..84592:2 2016-09-03/NB0387 85084..85102:2 Fringe: 2015-11-12/NB0921 45112..45116:2^45120^45122^45126^45128^45134^45136^45142..45154:2^46978^46980^47002..47024:2^47028..47032:2^47036 2016-01-09/HSC-Y 51008..51020:4^51022..51072:2^51078..51090:4^51092..51130:2^51136..51146:2^51150..51158:4^51160..51188:2^51192..51220:4^51222..51306:2 2016-01-11/NB0921 51762..51782:2 2016-02-03/HSC-Y 53808^53812^53818^53820^53824^53828^53844..53852:4^53860^53872^53874..53884:2^53888..53904:4^53906..53968:2^53972..54000:4^54002..54118:2 2016-03-14/HSC-Y 62518^62530^62532^62534^62542..62546:2^62572^62574^62578^62582^62584 2016-03-15/HSC-Y 63002..63098:2 2016-04-07/NB0921 67406..67432:2^67436^67438 2016-04-08/NB0921 67530..67544:2^67548..67554:2 2016-04-09/NB0921 67768..67800:2 2016-04-11/NB0921 68302..68332:2 2016-04-14/HSC-Y 69148..69240:2 2016-04-15/HSC-Y 69294..69300:2^69304^69308^69310^69312^69318..69326:2^69338^69364^69366..69398:2^69402..69422:2^69430..69524:2^69528..69596:2 Using LSST stack with w_2017_1 tag. . bias.sh root WARN: No data found for dataId=OrderedDict([('visit', 902642)]) root WARN: No data found for dataId=OrderedDict([('visit', 902644)]) root WARN: No data found for dataId=OrderedDict([('visit', 902646)]) root WARN: No data found for dataId=OrderedDict([('visit', 902648)]) root WARN: No data found for dataId=OrderedDict([('visit', 902650)]) root WARN: No data found for dataId=OrderedDict([('visit', 902652)]) root WARN: No data found for dataId=OrderedDict([('visit', 902654)]) root WARN: No data found for dataId=OrderedDict([('visit', 902656)]) root WARN: No data found for dataId=OrderedDict([('visit', 902658)]) root WARN: No data found for dataId=OrderedDict([('visit', 902660)]) root WARN: No data found for dataId=OrderedDict([('visit', 902662)]) root WARN: No data found for dataId=OrderedDict([('visit', 902664)]) root WARN: No data found for dataId=OrderedDict([('visit', 902666)]) root WARN: No data found for dataId=OrderedDict([('visit', 902668)]) root WARN: No data found for dataId=OrderedDict([('visit', 902670)]) root WARN: No data found for dataId=OrderedDict([('visit', 902672)]) root WARN: No data found for dataId=OrderedDict([('visit', 902674)]) root WARN: No data found for dataId=OrderedDict([('visit', 902676)]) root WARN: No data found for dataId=OrderedDict([('visit', 902678)]) root WARN: No data found for dataId=OrderedDict([('visit', 904542)]) root WARN: No data found for dataId=OrderedDict([('visit', 904544)]) root WARN: No data found for dataId=OrderedDict([('visit', 904546)]) root WARN: No data found for dataId=OrderedDict([('visit', 904548)]) root WARN: No data found for dataId=OrderedDict([('visit', 904550)]) root WARN: No data found for dataId=OrderedDict([('visit', 904552)]) root WARN: No data found for dataId=OrderedDict([('visit', 904554)]) root WARN: No data found for dataId=OrderedDict([('visit', 904556)]) root WARN: No data found for dataId=OrderedDict([('visit', 904558)]) root WARN: No data found for dataId=OrderedDict([('visit', 904560)]) root WARN: No data found for dataId=OrderedDict([('visit', 904562)]) root WARN: No data found for dataId=OrderedDict([('visit', 904564)]) root WARN: No data found for dataId=OrderedDict([('visit', 904566)]) root WARN: No data found for dataId=OrderedDict([('visit', 906234)]) root WARN: No data found for dataId=OrderedDict([('visit', 906236)]) root WARN: No data found for dataId=OrderedDict([('visit', 906238)]) root WARN: No data found for dataId=OrderedDict([('visit', 906240)]) root WARN: No data found for dataId=OrderedDict([('visit', 906242)]) root WARN: No data found for dataId=OrderedDict([('visit', 906244)]) root WARN: No data found for dataId=OrderedDict([('visit', 906246)]) root WARN: No data found for dataId=OrderedDict([('visit', 906248)]) root WARN: No data found for dataId=OrderedDict([('visit', 906250)]) root WARN: No data found for dataId=OrderedDict([('visit', 906252)]) I guess we pulled the pre-SSP engineering observations out of our data repo. Ignore them for now. bias-17 has: lsst::pex::exceptions::NotFoundError: 'Unable to find filter NB0468' Added that filter and NB0926 in obs_subaru; now using tickets/DM-8913 of obs_subaru. Reran bias-17. pprice@tiger-sumire:/tigress/pprice/dm-8913 $ find /tigress/HSC/HSC/CALIB-LSST-20170105 -name "BIAS-*-049.fits" | wc -l 17 Good! ingestCalibs.py /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB-LSST-20170105 '/tigress/HSC/HSC/CALIB-LSST-20170105/BIAS/*/NONE/BIAS-*.fits' --validity 180 rm -r /tigress/HSC/HSC/rerun/price/dm-8913/ . dark.sh dark-1: root WARN: No data found for dataId=OrderedDict([('visit', 904318)]) root WARN: No data found for dataId=OrderedDict([('visit', 904320)]) root WARN: No data found for dataId=OrderedDict([('visit', 904322)]) root WARN: No data found for dataId=OrderedDict([('visit', 904324)]) root WARN: No data found for dataId=OrderedDict([('visit', 904326)]) root WARN: No data found for dataId=OrderedDict([('visit', 904328)]) dark-18: dark.isr INFO: Performing ISR on sensor DataId(initialdata={'taiObs': '2016-08-25', 'pointing': 1698, 'visit': 83842, 'dateObs': '2016-08-25', 'filter': 'HSC-R2', 'field': 'DARK', 'ccd': 43, 'expTime': 300.0}, tag=set([])) lsst::pex::exceptions::RuntimeError: 'gsl_interp_init failed: invalid argument supplied by user [4]' Can't figure out what's wrong, so drop visit=83842. pprice@tiger-sumire:/tigress/pprice/dm-8913 $ find /tigress/HSC/HSC/CALIB-LSST-20170105 -name "DARK-*-049.fits" | wc -l 17 Good! ingestCalibs.py /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB-LSST-20170105 '/tigress/HSC/HSC/CALIB-LSST-20170105/DARK/*/NONE/DARK-*.fits' --validity 180 . flat.sh root WARN: No data found for dataId=OrderedDict([('visit', 903036)]) root WARN: No data found for dataId=OrderedDict([('visit', 903038)]) root WARN: No data found for dataId=OrderedDict([('visit', 903040)]) root WARN: No data found for dataId=OrderedDict([('visit', 903042)]) root WARN: No data found for dataId=OrderedDict([('visit', 903044)]) root WARN: No data found for dataId=OrderedDict([('visit', 903444)]) root WARN: No data found for dataId=OrderedDict([('visit', 903446)]) root WARN: No data found for dataId=OrderedDict([('visit', 903448)]) root WARN: No data found for dataId=OrderedDict([('visit', 903450)]) root WARN: No data found for dataId=OrderedDict([('visit', 903452)]) root WARN: No data found for dataId=OrderedDict([('visit', 902694)]) root WARN: No data found for dataId=OrderedDict([('visit', 902696)]) root WARN: No data found for dataId=OrderedDict([('visit', 902698)]) root WARN: No data found for dataId=OrderedDict([('visit', 902700)]) root WARN: No data found for dataId=OrderedDict([('visit', 902702)]) root WARN: No data found for dataId=OrderedDict([('visit', 902704)]) root WARN: No data found for dataId=OrderedDict([('visit', 904524)]) root WARN: No data found for dataId=OrderedDict([('visit', 904526)]) root WARN: No data found for dataId=OrderedDict([('visit', 904528)]) root WARN: No data found for dataId=OrderedDict([('visit', 904530)]) root WARN: No data found for dataId=OrderedDict([('visit', 904532)]) root WARN: No data found for dataId=OrderedDict([('visit', 904748)]) root WARN: No data found for dataId=OrderedDict([('visit', 904750)]) root WARN: No data found for dataId=OrderedDict([('visit', 904752)]) root WARN: No data found for dataId=OrderedDict([('visit', 904754)]) root WARN: No data found for dataId=OrderedDict([('visit', 904756)]) root WARN: No data found for dataId=OrderedDict([('visit', 904758)]) root WARN: No data found for dataId=OrderedDict([('visit', 904760)]) root WARN: No data found for dataId=OrderedDict([('visit', 904762)]) root WARN: No data found for dataId=OrderedDict([('visit', 904764)]) root WARN: No data found for dataId=OrderedDict([('visit', 904766)]) root WARN: No data found for dataId=OrderedDict([('visit', 906036)]) root WARN: No data found for dataId=OrderedDict([('visit', 906038)]) root WARN: No data found for dataId=OrderedDict([('visit', 906040)]) root WARN: No data found for dataId=OrderedDict([('visit', 906042)]) root WARN: No data found for dataId=OrderedDict([('visit', 906044)]) root WARN: No data found for dataId=OrderedDict([('visit', 906046)]) root WARN: No data found for dataId=OrderedDict([('visit', 906048)]) root WARN: No data found for dataId=OrderedDict([('visit', 906050)]) Whoops! Everything is NAN. Traced this to a failure to get the dark time (following the addition of VisitInfo). Fixed in obs_base branch tickets/DM-8962. Re-run the darks and the flats. flat-24: flat-28: flat-33: flat-34: flat-35: flat-38: lsst::pex::exceptions::RuntimeError: 'gsl_interp_init failed: invalid argument supplied by user [4]' pprice@tiger-sumire:/tigress/pprice/dm-8913 $ python -m pdb `which processCcd.py` /tigress/HSC/HSC --rerun price/dm-8913 --id visit=69882 ccd=43 -c doCalibrate=False charImage.doMeasurePsf=False charImage.doDeblend=False charImage.doApCorr=False --doraise pprice@tiger-sumire:/tigress/pprice/dm-8913 $ python -c 'import pickle, lsst.afw.math; lsst.afw.math.makeInterpolate(*pickle.load(open("args.pickle")))' #1 0x00002aaadf6ccae4 in lsst::afw::math::InterpolateGsl::InterpolateGsl ( this=0x6f4ef0, x=std::vector of length 30, capacity 32 = {...}, y=std::vector of length 30, capacity 32 = {...}, style=lsst::afw::math::Interpolate::AKIMA_SPLINE) at src/math/Interpolate.cc:211 211 int const status = ::gsl_interp_init(_interp, &x[0], &y[0], _y.size()); (gdb) p x $19 = std::vector of length 30, capacity 32 = {-4.6668978729026289, -1.0006934865900383, -2.2712418300653598, -0.76676245210727956, -0.70019157088122619, -0.63362068965517238, -0.5668103448275863, -0.5, -0.43342911877394619, -0.36685823754789276, -0.30028735632183906, -0.23371647509578544, -0.1669061302681992, -0.10009578544061301, -0.033524904214559385, 0.033045977011494247, 0.099616858237547928, 0.16618773946360157, 0.23299808429118776, 0.29980842911877403, 0.36637931034482762, 0.43295019157088127, 0.49952107279693497, 0.56609195402298851, 0.6329022988505747, 0.69971264367816088, 0.7662835249042147, 0.83285440613026807, 0.89942528735632199, 0.96623563218390807} Vector should be monotonic increasing, but third element is out of place. Fixed in tickets/DM-8982 of ip_isr. Great, I can re-run dark-18, including the frame that was causing trouble. Then I've gotta re-run flat-35 through 41 because those are affected, plus anything else that had gsl problems. Let's just re-run everything from 24 onwards. Done --- they succeed now. ingestCalibs.py /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB-LSST-20170105 '/tigress/HSC/HSC/CALIB-LSST-20170105/FLAT/*/*/FLAT-*.fits' --validity 180 rm -r /tigress/HSC/HSC/rerun/price/dm-8913/ . fringe.sh File "/tigress/pprice/dm-8913/pipe_drivers/python/lsst/pipe/drivers/constructCalibs.py", line 855, in processSingle self.subtractBackground.run(exposure) AttributeError: 'FringeTask' object has no attribute 'subtractBackground' Fixed in tickets/DM-8913 of pipe_drivers. ingestCalibs.py /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB-LSST-20170105 '/tigress/HSC/HSC/CALIB-LSST-20170105/FRINGE/*/*/FRINGE-*.fits' --validity 180 cp -r /tigress/HSC/HSC/CALIB-LSST-20160419/BFKERNEL /tigress/HSC/HSC/CALIB-LSST-20170105/ -------------------------------------------------------------------------------- Adding calibs for engineering data from the previous set of calibs: pprice@tiger-sumire:/tigress/HSC/HSC/CALIB-LSST-20170105/BIAS $ ln -s ../../CALIB-LSST-20160419/BIAS/2013-06-16 pprice@tiger-sumire:/tigress/HSC/HSC/CALIB-LSST-20170105/BIAS $ ln -s ../../CALIB-LSST-20160419/BIAS/2013-11-03 pprice@tiger-sumire:/tigress/HSC/HSC/CALIB-LSST-20170105/BIAS $ ln -s ../../CALIB-LSST-20160419/BIAS/2014-02-02 pprice@tiger-sumire:/tigress/HSC/HSC/CALIB-LSST-20170105/DARK $ ln -s ../../CALIB-LSST-20160419/DARK/2013-11-03 pprice@tiger-sumire:/tigress/HSC/HSC/CALIB-LSST-20170105/FLAT $ ln -s ../../CALIB-LSST-20160419/FLAT/2013-06-16 pprice@tiger-sumire:/tigress/HSC/HSC/CALIB-LSST-20170105/FLAT $ ln -s ../../CALIB-LSST-20160419/FLAT/2013-06-17 ingestCalibs.py /tigress/HSC/HSC --calib /tigress/HSC/HSC/CALIB-LSST-20170105 '/tigress/HSC/HSC/CALIB-LSST-20170105/BIAS/{2013-06-16,2013-11-03,2014-02-02}/NONE/*.fits' '/tigress/HSC/HSC/CALIB-LSST-20170105/DARK/2013-11-03/NONE/*.fits' '/tigress/HSC/HSC/CALIB-LSST-20170105/FLAT/2013-06-{16,17}/*/*.fits' --validity 180