It must be Shelley

. . . 'cause Blake don't shake like that.

For many a decade, I've been aware of Percy Bysshe Shelley's essay on atheism that got him kicked out of university—The Necessity of Atheism. Oddly, I don't remember reading the essay itself. Nor was I aware of Shelley's other essays on religious topics. (His essays are collected in separate volumes from his poetry, at least the ones I have.) His key essays reflecting his heterodoxy are available online:

Selected Prose Works of Shelley,
including, inter alia:
The Necessity of Atheism
A Refutation of Deism
On Life
On a Future State
Essay on Christianity

I discovered this in my search for "A Refutation of Deism" (1814).

Prometheus Books has collected these five essays in a book:

The Necessity of Atheism, and Other Essays (1993).

This collection of essays is available via Project Gutenberg:

A Defence of Poetry and Other Essays

Essays:
On Love
On Life in a Future State
On the Punishment of Death
Speculations on Metaphysics
Speculations on Morals
On the Literature, the Arts and the Manners of the Athenians
On the Symposium, Or Preface to the Banquet of Plato
A Defence of Poetry

Only the essay "On Life" is one of the key anti-religious tracts listed previously.

Offline the most comprehensive compilation of Shelley's prose is:

Shelley's Prose, edited by David Lee Clark (Albuquerque: University of New Mexico Press, 1954).

Contents:
Introduction:
The Growth of Shelley's Mind 3
Essays:
The Necessity of Atheism 37
An Address to the Irish People 39
Proposals for an Association of Philanthropists 60
A Declaration of Rights 70
A Letter to Lord Ellenborough 72
A Vindication of Natural Diet 81
Essay on the Vegetable System of Diet 91
"There Is No God" 97
" I Will Beget a Son" 103
"Necessity! Thou Mother of the World!" 109
"And Statesmen Boast of Wealth" 113
"Even Love Is Sold" 115
A Refutation of Deism 118
A Fragment of "A Refutation of Deism" 138
Refutation of the Christian Religion 141
A Fragment on Miracles 143
The Assassins 144
Essay on the Punishment of Death 154
A Proposal for Putting Reform to the Vote Throughout the Kingdom 158
An Address to the People on the Death of the Princess Charlotte 162
Essay on Love 169
Essay on Life 171
Essay on a Future State 175
Essay on the Revival of Literature 179
A Treatise on Morals 181
The Elysian Fields: A Lucianic Fragment 194
Essay on Christianity 196
Essay on Marriage 215
A Discourse on the Manners of the Ancient Greeks Relative to the Subject of Love 216
The Colosseum 224
A Philosophical View of Reform 229
Two Fragments on Reform 261
A System of Government by Juries 262
Essay on the Devil and Devils 264
A Defence of Poetry 275
Una Favola 298
Appendixes:
A. Literary Criticism 303
B. Prefaces to Poems 314
C. Fragments and Minor Pieces 337
D. Translations of Longer Foreign Language Passages 354
Bibliography:
Selected Bibliography 365
Index 371

This book contains some relevant items I've not found online. I put this fragment on my web site and perhaps will add some more material:

On Polytheism (1819?)

Of course, Shelley's poetry is not to be neglected, and all of it can be found online:

The Complete Poetical Works of Percy Bysshe Shelley

Other online texts by Shelley can be also be referenced via Project Gutenberg:

Shelley, Percy Bysshe - Project Gutenberg

Scholarly materials abound. These are the best web sites on English Romanticism:

Romantic Circles
Romanticism On the Net