How can BackgroundService be used safely for long-running processing in ASP.NET Core applications?

ASP.NET Core BackgroundService provides a framework for running background processing without blocking HTTP request execution. However, long-running workloads introduce important considerations around dependency lifetimes, cancellation, exceptions, concurrency, graceful shutdown, retries, and resource management. Simply placing an infinite loop inside a hosted service can create reliability and operational problems. How should developers design BackgroundService implementations that remain safe, observable, scalable, and resilient in production environments?