Fine Tune Your Polling and Batching in Mule ESB

2016-07-26 02:13:32

They say it's best to learn from others. With that in mind, let's dive into a use case I recently ran into. We were dealing with a number of legacy systems when our company decided to shift to a cloud-based solution. Of course, we had to prepare for the move — and all the complications that came with it.Use CaseWe have a legacy system built with Oracle DB using Oracle forms to create applications and lots and lots of stored procedures in the database. It's also been in use for over 17 years now with no major upgrades or changes. Of course, there have been a lot of development changes over these 17 years that taken the system close to the breaking point and almost impossible to implement something new. So, the company decided to move to CRM (Salesforce) and we needed to transfer data to SF from our legacy database. However, we couldn't create or make any triggers on our database to send real-time data to SF during the transition period.SolutionSo we decided to use Mule Poll to poll our database and get the records in bulk, then send them to SF using the Salesforce Mule connector.I am assuming that we all are clear about polling in general. If not, please refer to references at the end. Also, if you are not clear with Mule polling implementation there are few references at the bottom, too. Sounds simple enough doesn't it? But wait, there are few things to consider.What is the optimum timing of the poll frequency of your polls?How many threads of each poll you want to have? How many active or inactive threads do you want to keep?.How many polls can we write before we break the object store and queue store used by Mule to maintain your polling?What is the impact on server file system if you use watermark values of the object store?How many records can we fetch in one query from the database?How many records can we actually send in bulk to Salesforce using SFDC?These are few, if not all the considerations you have to do before implementation. The major part of polling is the WATERMARK of polling and how Mule implements the watermark in the server.Polling for Updates Using WatermarksRather than polling a resource for all its data with every call, you may want to acquire only the data that has been newly created or updated since the last call. To acquire only new or updated data, you need to keep a persistent record of either the item that was last processed, or the time at which your flow last polled the resource. In the context of Mule flows, this persistent record is called a watermark.To achieve the persistency of watermark, Mule ESB will store the watermarks in the object store of the runtime directory of a project in the ESB server. Depending on the type of object store you have implemented, you may have a SimpleMemoryObjectStore or TextFileObjectStore, which can be configured like below: Below is a simple memory object store sample: Below is text file object store sample: For any kind of object store, Mule ESB creates files in-server, and if the frequency of your polls are not carefully configured, then you may run into file storage issues on your server. For example, if you are running your poll every 10 seconds with multiple threads, and your flow takes more than 10 seconds to send data to SF, then a new object store entry is made to persist the watermark value for each flow trigger, and we will end up with too many files in the server object store.To set these values, we have consider how many records we are fetching from the database, as SF has limit of 200 records that you can send in one bulk. So, if you are fetching 2,000 records, then one batch will call SF 10 times to transfer  these 2,000 records. If your flow takes five seconds to process 200 records, including the network transfer to send data to SF and come back, then your complete poll will take around 50 seconds to transfer 2,000 records.If our polling frequency is 10 seconds, it means we are piling up the object store.Another issue that will arise is the queue store. Because the frequency and execution time have big gaps, the queue store's will also keep queuing. Again, you have to deal with too many files.To resolve this, it’s always a good idea to fine-tune your execution time of the flow and frequency to keep the gap small. To manage the threads, you can use Mule's batch flow threading function to control how many threads you want to run and how many you want to keep active.I hope few of the details may help you set up your polling in a better way.There are few more things we have to consider. What happens when error occurs while sending data? What happens when SF gives you error and can't process your data? What about the types of errors SF will send you? How do you rerun your batch with the watermark value if it failed? What about logging and recovery? I will try to cover these issues in a second blog post.Refrences:https://docs.mulesoft.com/mule-user-guide/v/3.6/poll-reference#polling-for-updates-using-watermarkshttps://docs.mulesoft.com/mule-user-guide/v/3.7/poll-referencehttps://docs.mulesoft.com/mule-user-guide/v/3.7/poll-schedulers#fixed-frequency-schedulerhttps://en.wikipedia.org/wiki/Polling_(computer_science)

SpaceX rocket lifts off on cargo run, then lands at launch site

2016-07-19 06:08:56

CAPE CANAVERAL, Fla. An unmanned SpaceX rocket blasted off from Florida early on Monday to send a cargo ship to the International Space Station, then turned around and landed itself back at the launch site.The 23-story-tall Falcon 9 rocket, built and flown by Elon Musk’s Space Exploration Technologies, or SpaceX, lifted off from Cape Canaveral Air Force Station at 12:45 a.m. EDT (0445 GMT).Perched on top of the rocket was a Dragon capsule filled with nearly 5,000 pounds (2,268 kg) of food, supplies and equipment, including a miniature DNA sequencer, the first to fly in space.Also aboard the capsule was a metal docking ring of diameter 7.8 feet (2.4 m), that will be attached to the station, letting commercial spaceships under development by SpaceX and Boeing Co. ferry astronauts to the station, a $100-billion laboratory that flies about 250 miles (400 km) above Earth. The manned craft are scheduled to begin test flights next year.Since NASA retired its fleet of space shuttles five years ago, the United States has depended on Russia to ferry astronauts to and from the station, at a cost of more than $70 million per person.As the Dragon cargo ship began its two-day journey to the station, the main section of the Falcon 9 booster rocket separated and flew itself back to the ground, touching down a few miles south of its seaside launch pad, accompanied by a pair of sonic booms. "Good launch, good landing, Dragon is on its way," said NASA mission commentator George Diller.Owned and operated by Musk, the technology entrepreneur who founded Tesla Motors Inc, SpaceX is developing rockets that can be refurbished and re-used, potentially slashing launch costs. With Monday’s touchdown, SpaceX has successfully landed Falcon rockets on the ground twice and on an ocean platform during three of its last four attempts.SpaceX intends to launch one of its recovered rockets as early as this autumn, said Hans Koenigsmann, the firm's vice president for mission assurance. (Reporting by Irene Klotz, Editing by Chris Michaud and Clarence Fernandez)

Tesla crash raises stakes for self-driving vehicle startups

2016-07-12 09:34:19

DETROIT/SAN FRANCISCO Concerns raised by the first reported fatality in a semi-automated car were expected to speed adoption of more sensitive technology to help vehicles see and drive themselves safely, increasing demand on the emerging autonomous vehicle technology industry, investors and analysts said.Goldman Sachs forecasts the market for advanced driver assistance systems and autonomous vehicles will grow from about $3 billion last year to $96 billion in 2025 and $290 billion in 2035. More than half of that revenue in 20 years, Goldman estimates, will come from radar, cameras and lidar, a sensor that uses laser – all tools considered essential to building vehicles that can pilot themselves.The May 7 death of Ohio technology company owner Joshua Brown in a Tesla Motors Inc (TSLA.O) Model S while the car's semi-automated Autopilot system was engaged highlighted the limitations of current automated driving systems.Tesla’s Autopilot system uses cameras and radar, but not lidar. The company said its system would have had trouble distinguishing a white semi-trailer positioned across a road against a bright sky.Industry executives and analysts told Reuters they expect the Tesla crash will spur investment in self-driving vehicle systems that combine multiple of sensors, including lidar."As we move to a higher level of autonomy in vehicles, you’re going to want to have more redundancy," which radar and lidar can provide, Dan Galves, senior vice president at vision safety system maker Mobileye NV(MBLY.N) , said in an interview. "The more sensors, the better."Carmakers have been using multiple sensors in prototypes that are in testing but not yet ready for market. A variety of technologies with overlapping capabilities is seen as a way to increase safety under a wider range of circumstances.The valuations of some self-driving startups "may even increase if there are companies that can solve some of the issues" the Tesla accident raised, said Quin Garcia, managing director of AutoTech Ventures, a Silicon Valley investment firm.Semi-automated systems such as General Motor Co's (GM.N) SuperCruise and Traffic Jam Pilot from Volkswagen AG's (VOWG_p.DE) Audi are due on the market in 2017-2018. Ford Motor Co(F.N) expects to deploy a semi-automated system, using Velodyne lidar, in 2018. Toyota Motor Corp(7203.T), which is investing more than $1 billion in such self-driving technologies as robotics and artificial intelligence, said it aims to put fully driverless cars on the road in time for the 2020 Tokyo Olympics.Delphi Automotive PLC (DLPH.N) plans to build lidar vision systems with technology from Quanergy Systems, which makes solid state lidar systems. Delphi plans to combine information from the lidar system with radar and other driver assistance technology to create a 360-degree view around a car, a company official said. Delphi has an investment in Quanergy, one of more than 50 self-driving startups that together have raised more than $800 million in investment capital in the past decade, according to a Reuters analysis of publicly available data.At least two of those startups - Quanergy which makes solid state lidar sensors, and Zoox, which is developing fully automated vehicle systems - have jumped in value to more than $1 billion each since GM's $1.2-billion acquisition earlier this year of another self-driving startup, Cruise Automation.Quanergy and Zoox hope to follow the lead of Mobileye, an Israeli supplier of vision-based safety systems to 25 global automakers, including Tesla. Co-founded in 1999 by a computer science professor at Hebrew University, Mobileye went public in 2014 and today is valued at nearly $10 billion.Mobileye plans by 2020 to offer a hardware/software system that can gather, fuse and analyze data from 20 different sensors, including cameras, lidar and radar. The company's new EyeQ5 "system on chip" will be a key component in a fully autonomous driving system that is being jointly developed with BMW AG (BMWG.DE) and Intel Corp (INTC.O) and is aimed at production in 2021.Like Mobileye, Velodyne, a leading supplier of laser-based lidar systems, works with many of the world’s top automakers, including Ford, GM, BMW, Toyota Honda Motor Co(7267.T) and Daimler AG’s (DAIGn.DE) Mercedes-Benz."Our clients want to (combine) lidar and cameras," Velodyne's Marta Hall, president of business development, told Reuters in an interview. Automakers are stepping up orders as lidar systems come down in size and price, she said. Among the potential beneficiaries of this growing interest is LeddarTech, a relatively young startup based in Canada's Quebec City. The company is providing LED-based lidar systems to French supplier Valeo (VLOF.PA), which also buys vision-based systems from Mobileye.Germany's Robert Bosch, which has been developing self-driving components and systems for more than 15 years, buys lidar from an unnamed Tier II supplier and intends to package it in a highly automated “highway pilot” system intended for series production in 2020, said spokesman Tim Wieland."Bosch sees the necessity for a sensor setup that includes radar, video and lidar," Wieland said. The three sensors "complement each other very efficiently."REGULATION AND LITIGATION WILD CARDS Regulation and litigation are two big wild cards for the autonomous driving sector.Safety regulators and industry executives have said self-driving cars ultimately could slash traffic fatalities – about 35,000 last year in the United States and more than 1.2 million globally - by up to 90 percent. But regulators are also concerned that drivers could be lulled into unsafe behavior by systems that take control for a time, but expect human operators to re-take command in an emergency.The National Highway Traffic Safety Administration is investigating the role of Autopilot in the Florida accident and another crash in Pennsylvania involving a Tesla vehicle. The agency also is expected to roll out this summer broad guidelines for deploying autonomous vehicle technology."I hope NHTSA does not overreact" to the crash, said Stefan Heck, co-founder of Nauto, another Silicon Valley self-driving startup with corporate backing. "The tradeoff is quite clear: Some safety improvement is better than none."Product liability for automated vehicles is uncharted territory. The U.S. Transportation Department has said an automated driving system could be considered the "driver" for regulatory purposes.Industry executives are betting that consumer interest in the technology will rise.A survey conducted by AlixPartners in June - before the Tesla accident was reported publicly - found that 90 percent of respondents would be interested in a self-driving car that would let the driver take the wheel from time to time. The same survey noted that nearly 80 percent of respondents would pay for the technology - including 10 percent who would spend up to $5,000.The favorable response rates are much higher than in previous surveys on self-driving technology.News of the Tesla crash "is not going to put too much of a dent in public perception" of self-driving cars, said AlixPartners' Mark Wakefield. (Reporting by Paul Lienert in Detroit and Alexandria Sage in San Francisco; Editing by Joe White and Lisa Girion)

NASA's Juno spacecraft ready for one-shot try to orbit Jupiter

2016-07-04 23:09:19

CAPE CANAVERAL, Fla. A NASA spacecraft was poised for a one-shot attempt to slip into Jupiter's orbit on Monday for the start of a 20-month-long dance around the solar system's largest planet to learn how and where it formed.Flight controllers at NASA's Jet Propulsion Laboratory in Pasadena, California, were preparing for a long night as the Juno probe streaked closer toward Jupiter at 200 times the speed of sound in the empty vacuum of space."We're barreling down," Juno lead scientist Scott Bolton of the Southwest Research Institute in San Antonio told reporters on Monday.By noon on Monday, Juno had sailed past three of Jupiter's four main moons, with volcanic Io, the innermost big moon, in its sights.Confirmation of whether Juno, the only solar-powered spacecraft ever dispatched to the outer solar system, had successfully placed itself into polar orbit around Jupiter was not expected until 11:53 p.m. EDT on Monday (0353 GMT on Tuesday).Launched from Florida nearly five years ago, Juno must be precisely positioned, ignite its main engine at exactly the right time and keep it burning for 35 minutes to shed enough speed so it can be captured by Jupiter's gravity. If anything goes even slightly awry, Juno will sail helplessly past Jupiter, unable to complete a $1 billion mission to peer through the planet's thick atmosphere and map its gargantuan magnetic field.Scientists are particularly interested in learning how much water Jupiter contains, which is key to determining where in the solar system it formed. Jupiter's origins, in turn, affected the development and position of the rest of the planets, including Earth and its fortuitous location conducive to the evolution of life.The immense gravity exerted by Jupiter's sheer size - packing 2-1/2 times the mass of all the other planets combined - is thought to have helped shield Earth from bombardment by comets and asteroids. "We are learning about nature, how Jupiter formed and what that tells us about our history and where we came from," Bolton said.The Juno probe is named for the ancient Roman goddess, who was the wife and sister of Jupiter, the mythological king of gods, and had the power to see through clouds.'MUSICAL NOTES' Only one other spacecraft, Galileo, has ever circled Jupiter, which is five times farther away from the sun than Earth and is itself orbited by 67 known moons. Bolton said Juno is likely to discover even more.Seven other U.S. space probes have sailed past the gas giant on brief reconnaissance missions before heading elsewhere in the solar system.Ground control teams will monitor Juno's progress during its do-or-die engine burn by listening for a series of radio signals."They really are musical notes. Based on what musical note is sent, we will know how something is doing," Bolton said.During its approach, Juno also must be lucky enough to fly through Jupiter's tenuous rings without being hit by particles zipping around so fast that even a speck the size of a blood cell could prove fatal. The risks to the spacecraft will not end once it arrives in orbit. The probe must quickly turn around and face the sun so its 18,698 solar cells can begin recharging the battery."I won't exhale until we are back sun-pointing again," Bolton said.Juno will fly in highly elliptical, egg-shaped orbits that pass within 3,000 miles (4,800 km) of the tops of Jupiter's clouds and inside the planet's powerful radiation belts.Juno's computers and sensitive science instruments are housed in a 400-pound (180-kg) titanium vault for protection. But during its 37 orbits around Jupiter, Juno will be exposed to the equivalent of 100 million dental X-rays, said Bill McAlpine, radiation control manager for the mission.The spacecraft, built by Lockheed Martin, is expected to last for 20 months. On its final orbit, Juno will dive into Jupiter's atmosphere, where it will be crushed and vaporized.Like Galileo, which circled Jupiter for eight years before crashing into the planet in 2003, Juno's demise is designed to prevent any hitchhiking microbes from Earth from inadvertently contaminating Jupiter's ocean-bearing moon Europa, a target of future study for extraterrestrial life. (Editing by Peter Cooney and Sandra Maler)

Google beats children's web privacy appeal, Viacom to face one claim

2016-06-27 22:54:36

Google and Viacom on Monday defeated an appeal in a nationwide class action lawsuit by parents who claimed the companies illegally tracked the online activity of children under the age of 13 who watched videos and played video games on Nickelodeon's website.By a 3-0 vote, the 3rd U.S. Circuit Court of Appeals in Philadelphia said Google, a unit of Alphabet Inc, and Viacom Inc were not liable under several federal and state laws for planting "cookies" on boys' and girls' computers, to gather data that advertisers could use to send targeted ads.The court also revived one state law privacy claim against Viacom, claiming that it promised on the Nick.com website not to collect children's personal information, but did so anyway.Monday's decision largely upheld a January 2015 ruling by U.S. District Judge Stanley Chesler in Newark, New Jersey. It returned the surviving claim to him.Jay Barnes, a lawyer for the parents, declined to comment.Viacom spokesman Jeremy Zweig said the company is pleased with the dismissals and confident it will prevail on the remaining claim. "Nickelodeon is proud of its record on children's privacy issues and strongly committed to the best practices in the industry," he added. Google did not immediately respond to a request for comment.Monday's decision is a fresh setback for computer users, after the same appeals court last November 10 said Google was not liable under federal privacy laws for bypassing cookie blockers on Apple Inc's Safari browser and Microsoft Corp's Internet Explorer browser.Circuit Judge Julio Fuentes, who wrote both decisions, said that ruling doomed many of the parents' claims against Mountain View, California-based Google and New York-based Viacom. He also rejected the parents' claims under the Video Privacy Protection Act, a 1988 law adopted a year after a newspaper wrote about movies rented by failed Supreme Court nominee Robert Bork, based on a list provided by a video store.Fuentes said the law was meant to thwart the collection of data to help monitor people's video-watching behavior.He said Congress, despite amending the law in 2013, never updated it to cover the collection of data such as users' IP addresses, browser settings and operating settings, and reflect a "contemporary understanding" of Internet privacy. "Some disclosures predicated on new technology, such as the dissemination of precise GPS coordinates or customer ID numbers, may suffice," Fuentes wrote. "But others--including the kinds of disclosures described by the plaintiffs here--are simply too far afield from the circumstances that motivated the act's passage to trigger liability."The revived privacy claim accused Viacom of reneging on a promise on Nick.com that said: "HEY GROWN-UPS: We don't collect ANY personal information about your kids. Which means we couldn't share it even if we wanted to!"Fuentes said a reasonable jury might find Viacom liable for "intrusion upon seclusion" if it found its alleged privacy intrusion "highly offensive to the ordinary reasonable man."The case is In re: Nickelodeon Consumer Privacy Litigation, 3rd U.S. Circuit Court of Appeals, No. 15-1441. (Reporting by Jonathan Stempel in New York; Editing by David Gregorio; Editing by David Gregorio)

Older Post
Twitter lawsuit partly dismissed over U.S. information requests
Drug driving suit mimics taking the wheel stoned
GM adenovirus used by doctors to attack tumor cells